A method for constructing a sustainable labeling network dataset based on hybrid supervised learning

Abstract

This application discloses a method for constructing a network dataset. The method includes: receiving raw network data; processing the raw network data to obtain a standardized feature sequence; using an encoder to perform unified semantic encoding on the standardized feature sequence to obtain a semantic vector corresponding to each sample; based on the semantic vector, using a labeling model to pre-label the samples, generating temporary labels and their predicted confidence scores; based on the semantic vector, constructing a local semantic structure through clustering, performing clustering noise detection on the pre-labeled samples to determine semantic consistency; and combining predicted confidence scores and semantic consistency to perform risk-based triage of the pre-labeled samples, and handling various types of samples differently according to the triage results. This method supports continuously flowing multi-source network data, has a risk triage mechanism constrained by both semantic consistency and predicted confidence scores, and a closed-loop label governance mechanism for network datasets.

Description

Technical Field

[0001] This application relates to methods for constructing network datasets, and more particularly to a method for constructing sustainable labeled network datasets based on hybrid supervised learning. Background Technology

[0002] Network datasets are a crucial foundation for network security detection, network behavior measurement, abnormal traffic discovery, service identification, and model training. Network data includes, but is not limited to, raw PCAP packets (i.e., raw binary data packets captured from network links using packet capture software such as Wireshark), flow records aggregated by 5-tuples or sessions, NetFlow / IPFIX statistical records (i.e., standardized traffic statistics logs output by network devices; NetFlow is an earlier standard introduced by Cisco, while IPFIX (IP Flow Information Export) is a more general, standardized version), Zeek parsing logs (i.e., structured text logs output by layering various network protocols using the Zeek open-source traffic analysis engine), DNS / HTTP / TLS (Transport Layer Security Protocol) metadata, alarm records (e.g., alarm records generated by firewalls, Intrusion Detection Systems (IDS), and Intrusion Prevention Systems (IPS), device configuration and security event logs, etc. High-quality network datasets not only require a sufficient number of samples, but also require that the source of the samples, the processing flow, the semantics of the labels, the label evidence, and the subsequent iterative update process be continuously tracked and verified to ensure that each sample in the dataset has a traceable source, a verifiable process, and a labeling result that has undergone a complete label governance process.

[0003] Existing publicly available cybersecurity datasets can provide benchmarks for algorithm research and system evaluation, but they generally suffer from problems such as early collection time, fixed application scenarios, lagging updates to protocols and attack types, inconsistent labeling standards, and insufficient performance when migrating across real networks. With the rapid evolution of cloud computing, mobile terminals, encrypted communication, the Internet of Things, and new attack behaviors, static datasets built in one go often fail to reflect the characteristics of real network traffic in the long term and cannot support the continuous iterative training of detection models.

[0004] Existing network data annotation schemes can be broadly divided into two categories: one is the traditional single annotation scheme (such as manual annotation, rule annotation, and automatic model annotation), and the other is the collaborative annotation scheme that integrates supervised representation learning and expert review.

[0005] Traditional single annotation schemes each have their limitations: manual annotation has high reliability, but it requires the participation of experts with experience in network protocols, attack behavior, application identification, and log correlation. Processing large-scale samples one by one is time-consuming and time-consuming. Rule annotation is more efficient, but its coverage is limited by the expert's experience and rule maintenance capabilities. Automatic model annotation can improve throughput, but it is more susceptible to factors such as historical noisy labels, inflated prediction confidence, changes in sample class distribution, and insufficient coverage of weak rules, which can lead to the repeated propagation of incorrect labels during continuous training and data accumulation.

[0006] The synergy between weakly supervised label generation, semi-supervised learning, self-supervised representation learning, and expert review provides a technological foundation for reducing labeling costs. For example, weakly supervised label generation can abstract expert experience, heuristic rules, file paths, protocol fields, and external intelligence into labeling functions; self-supervised representation learning can learn a shared semantic space on unlabeled or existing weakly labeled samples; and expert review can provide final confirmation for samples with low prediction confidence, high risk, or semantic anomalies.

[0007] However, most existing collaborative annotation schemes only focus on single label generation or model classification accuracy optimization, failing to integrate multi-source network data access, hybrid supervised automatic pre-annotation, semantic consistency noise detection, expert review, dataset accumulation, and annotation model iteration into an integrated closed-loop label governance mechanism.

[0008] For example, Chinese invention patent application CN116684127A discloses a method for interpretable network data labeling for network security. This method uses an attack simulator to generate and capture data packets, then clusters them to construct a final dataset. Suspicious traffic is identified through interaction between anomaly detectors, interpreters, and network analysts, emphasizing the interpretability of the labeling. Chinese invention patent application CN116232699A discloses a training method for a fine-grained network intrusion detection model. This method trains an initial model using existing labeled samples, then generates temporary labels for unlabeled samples, and uses the predicted confidence level based on these temporary labels to select samples for further training.

[0009] Most of these existing technologies only support single-source network data processing and have limited support for mixed supervised annotation processes; the annotation process is geared towards offline fixed batch data construction, making it difficult to adapt to continuously flowing streaming network traffic; there is a lack of a balancing scheduling mechanism between automatic annotation and expert review, which can easily lead to excessive burden on manual review or over-reliance on automatic annotation and poor annotation accuracy; the label generation process lacks a closed-loop governance system, making it difficult to effectively distinguish and manage the status of labels generated in different ways. Summary of the Invention

[0010] To address one or more of the aforementioned issues, this application proposes a method and system for constructing a sustainable labeled network dataset based on hybrid supervised learning. This method performs preprocessing, unified semantic representation, automatic pre-labeling, semantic consistency noise detection, prediction confidence splitting, expert review, dataset accumulation, and annotation model iteration on continuously incoming network data. The output is a network dataset with label source, label status, version records, and expert review records. This construction method and system can perform unified semantic representation, risk sample splitting, and traceable label write-back on continuously incoming multi-source network data, even with only a small number of high-quality, strongly labeled samples, a limited number of expert rules, or existing weak labels. Furthermore, it ensures that verified labels, dataset versions, and annotation model versions can continuously serve subsequent sample labeling and model updates, thereby achieving automated, auditable, and sustainable dataset accumulation and realizing the construction of a sustainable labeled network dataset.

[0011] In a first aspect, this application proposes a method for constructing a network dataset, wherein the method includes: receiving raw network data, processing the raw network data to obtain a standardized feature sequence; using an encoder to perform unified semantic encoding on the standardized feature sequence to obtain a semantic vector corresponding to each sample; based on the semantic vector, using a labeling model to perform pre-labeling on the samples, generating temporary labels and their predicted confidence scores for the samples; based on the semantic vector, constructing a local semantic structure through clustering, performing clustering noise detection on the pre-labeled samples to determine semantic consistency, wherein semantic consistency characterizes the degree of consistency between the temporary labels and the local semantic structure; and combining the predicted confidence score and semantic consistency to perform risk diversion on the pre-labeled samples, and handling various types of samples differently according to the diversion results.

[0012] This technical solution constructs a unified semantic space through an encoder and synchronously achieves sample pre-labeling and cluster semantic consistency detection based on semantic vectors. It establishes a risk diversion mechanism constrained by both semantic consistency and prediction confidence, evaluating label reliability from two dimensions: model output confidence level and the local distribution of samples within the shared semantic space. This mechanism identifies high-risk samples whose labels deviate from local semantic centers and implements differentiated handling. This solution helps distinguish between high-confidence compliant samples and questionable noise samples, reducing the possibility of erroneous labels being directly included in the dataset and continuously accumulating and spreading. The entire process supports continuous inflow and automated processing of multi-source network data, reducing the need for expert review and improving label quality while enhancing dataset construction efficiency.

[0013] In some embodiments, the labeling model is a hybrid supervised labeling model, wherein the hybrid supervised labeling model integrates at least two types of supervision sources, uniformly calibrates the multi-source supervision information and outputs the category probability, thereby generating temporary labels corresponding to samples and their prediction confidence.

[0014] This hybrid supervised labeling model integrates multiple supervision sources to broaden traffic coverage scenarios, outputs calibrated prediction confidence after unified calibration, and coordinates conflicts between different supervision sources to reduce the possibility of pre-labeled noise accumulation.

[0015] In some embodiments, the construction method further includes retaining a full-process label governance record, which includes: upon receiving raw network data, simultaneously collecting raw network data source information; after extracting standardized feature sequences, constructing a data source ledger and initializing a label governance record for each sample; when performing pre-labeling on samples, writing pre-labeling information into the label governance record of the corresponding sample; when performing clustering noise detection on samples, writing noise detection information into the label governance record of the corresponding sample; when performing sample splitting, writing sample splitting information into the label governance record of the corresponding sample; and when handling different types of samples differently, writing handling result information into the sample's label governance record; wherein, the pre-labeling information includes at least temporary labels and predicted confidence levels, the noise detection information includes at least the reasons for label conflicts, the sample splitting information includes at least the splitting type, and the handling result information includes at least the confirmed labels. All the above-mentioned full-process label details are recorded in the sample-level label governance record, the data source ledger only stores the original sample collection summary and association pointer, and the data source ledger and the label governance record are associated through a unique sample identifier.

[0016] Building upon the previous work, this technical solution establishes a closed-loop label governance mechanism for network datasets, employing a hierarchical, interconnected storage architecture of a data source ledger and sample-level label governance records. Relying on unique sample identifiers, it links original data source information, pre-labeling results, semantic noise detection conclusions, sample distribution types, and final label handling, comprehensively retaining operational information at each stage of a sample's lifecycle. This ensures that the entire process of label formation, verification, distribution, and review for each sample is auditable and reproducible. The complete end-to-end log continuously provides feedback on labeling defects and noise sources, supporting iterative optimization of the labeling model and dynamic updates to the dataset. It records information about samples at each stage of label formation, verification, distribution, and review to support auditing and reproducibility of related processes and provide a basis for model updates.

[0017] In some embodiments, pre-labeled samples are split based on prediction confidence and semantic consistency, and differentiating the types of samples according to the splitting results. This includes: obtaining the probability output by the labeling model for each candidate label, and taking the highest prediction probability as the prediction confidence; using the K-Means clustering algorithm to cluster the semantic vectors of the samples to form a local semantic structure, calculating the distance from the semantic vector of a sample to the semantic vector of its cluster center to characterize semantic consistency, and comparing the temporary labels of samples within the same cluster to identify samples with label conflicts within the cluster; classifying the prediction confidence of samples into three categories—high prediction confidence, intermediate prediction confidence, or low prediction confidence—based on a first set of thresholds; classifying samples based on semantic consistency based on a second set of thresholds to obtain two results: semantically consistent or semantically abnormal; splitting the pre-labeled samples into three categories—automatic confirmation, cautious retention, and expert review—based on prediction confidence and semantic consistency for differentiated processing; and generating a continuously updated network dataset based on the differentiated samples.

[0018] This technical solution uses an integrated approach combining K-Means clustering with intra-cluster label conflict identification and dual-threshold hierarchical judgment to provide a standardized implementation method for identifying semantic consistency risks in network traffic samples. It comprehensively judges the risk of sample labeling from two independent dimensions: prediction confidence and semantic consistency, and realizes fine-grained hierarchical diversion of samples to screen suspicious samples with label noise. Furthermore, it constructs a continuously iteratively updated network dataset based on the differentiated processing results.

[0019] In some embodiments, when a sample has high prediction confidence and the temporary label is semantically consistent with the local semantic structure, the sample is diverted to automatic confirmation; when a sample has low prediction confidence or the temporary label is semantically abnormal with respect to the local semantic structure, the sample is diverted to expert review; when a sample does not fall into either of the above two categories, the sample is diverted to cautious retention.

[0020] This technical solution sets up non-overlapping stratified sample triage judgment conditions, clearly delineates the distinction boundaries between three types of samples: automatic confirmation, expert review, and cautious retention, avoids sample classification confusion, and achieves differentiated hierarchical management of samples with different risk levels.

[0021] In some embodiments, when a sample is assigned to automatic confirmation, the temporary label corresponding to the sample becomes the automatic confirmation label, and the automatic confirmation label is written to the dataset as a confirmed label; when a sample is assigned to expert review, the sample is sent to the expert review queue, the expert confirmation label is received from the experts, the review record is written to the sample's label governance record, and the expert confirmation label is written to the dataset as a confirmed label; and when a sample is assigned to cautious retention, the temporary label corresponding to the sample is retained to await verification in subsequent versions.

[0022] This technical solution provides differentiated processing logic for the three types of diverted samples. It adds a temporary verification buffer mechanism for cross-dataset version iteration for cautiously retained samples in the intermediate state (i.e., cautious retention), instead of sending them directly to automatic confirmation or expert review. This avoids noise caused by inaccurate labels and reduces the workload of expert review. The correction results generated by expert review are simultaneously stored in the label management record, and the labeling basis for manual intervention is fully preserved.

[0023] In some embodiments, samples in the expert review queue are sorted based on risk scores, wherein the risk scores are calculated based on risk score evaluation indicators, which include one or more of the following: entropy value, maximum prediction probability, vector distance from the sample to its cluster center, neighborhood label consistency rate, number of label function conflicts, sample source credibility, and historical review results.

[0024] This technical solution integrates multi-dimensional evaluation indicators such as entropy, cluster distance, and label conflict to uniformly calculate the sample risk score, enabling the expert review queue to be automatically and orderly scheduled according to the sample risk priority, giving priority to ensuring that high-risk suspicious samples are manually verified, and improving the utilization efficiency of expert review resources.

[0025] In some embodiments, the construction method further includes: updating at least one class of objects, including the encoder, the annotation model, a first set of thresholds, and a second set of thresholds; and for samples that are carefully retained, re-executing the processing steps affected by the update based on the object that has been updated, the processing including one or more of unified semantic encoding, pre-annotation, clustering noise detection, and splitting, thereby splitting the sample again based on prediction confidence and semantic consistency.

[0026] This technical solution supports selective re-evaluation of cautiously retained samples after any version update of the encoder, annotation model, or traffic splitting threshold, continuously achieving dynamic self-correction and optimization of the dataset. The encoder, annotation model, and threshold can all be independently iteratively adjusted, significantly improving the solution's adaptability and generalization capabilities to diverse network service scenarios. Specifically, the traffic splitting process relies solely on prediction confidence and semantic consistency for routing determination; risk scores are not involved in this routing decision. Risk scores are only used to prioritize samples within the expert review queue after they have been split, consistent with the previously described purpose of risk scores.

[0027] In some embodiments, the construction method further includes: updating the encoder, including: masking features at a portion of the time steps in the feature sequence and performing self-supervised pre-training on the encoder based on the reconstruction task of the masked content; updating the labeled model, including: training the labeled model using trusted samples, which include expert-confirmed samples and automatically confirmed samples; and updating the first set of thresholds and the second set of thresholds, including: dynamically adjusting the thresholds according to the model version, sample category distribution, expert workload, and business risk.

[0028] This technical solution defines three independent update logics: encoder self-supervised pre-training, training of labeled models using reliable samples, and dynamic threshold adjustment. This adapts to the optimization needs of different iteration stages of the dataset. The three update paths are independent yet can be combined to continuously optimize semantic representation accuracy and sample splitting criteria. Automatically confirmed samples used for training the labeled model should simultaneously meet three conditions: high prediction confidence, consistency between the label and local semantic structure, and passing quality sampling. When automatically confirmed samples participate in training, an upper limit is configured on the proportion of samples from a single class, and a mechanism for regular sample sampling and historical sample replay verification is implemented to suppress the risk of accumulated confirmation bias caused by automatic confirmation.

[0029] In a second aspect, this application proposes a system for constructing a network dataset, comprising: a data access and standardization module for receiving raw network data and processing it to obtain standardized feature sequences; a semantic representation module for using an encoder to perform unified semantic encoding on the standardized feature sequences to obtain a semantic vector corresponding to each sample; a pre-labeling module for performing pre-labeling on samples based on the semantic vector using a labeling model, wherein the labeling model integrates at least two types of supervision sources, uniformly calibrates the multi-source supervision information, and outputs class probabilities to generate temporary labels and their predicted confidence scores for the samples; a noise detection module for performing clustering noise detection on the pre-labeled samples based on the semantic vector, constructing a local semantic structure through clustering to determine semantic consistency, wherein semantic consistency characterizes the degree of consistency between the temporary labels and the local semantic structure; a confidence-based splitting module for splitting the pre-labeled samples into three categories: automatic confirmation, cautious retention, and expert review, based on the predicted confidence score and semantic consistency; and a splitting processing module for handling the various types of samples differently according to the splitting results. The system also includes a data accumulation and iterative update module, which generates continuously updated network datasets and retains full-process label governance records.

[0030] This system supports the continuous inflow of multi-source network data, and features a closed-loop label governance mechanism for network datasets and a risk diversion mechanism that combines semantic consistency and prediction confidence constraints, thereby enabling the construction of sustainably labeled network datasets.

[0031] In a third aspect, this application proposes a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the steps of the construction method described in the embodiments of this application.

[0032] The core function of this technical solution is not single-time network traffic classification, but rather the establishment of a repeatable closed-loop label governance system. Each batch of data entering the system records its source, processing status, and label evidence. The automation module only solidifies temporary labels into confirmed labels when the prediction confidence and semantic consistency conditions are met. For data with noise risks, rule conflicts, or low prediction confidence, the system diverts it to the expert review queue. The results of expert review and confirmation are then re-accumulated as trainable data (expert-confirmed samples) and associated with the labeled model version, data source ledger, and indicator records, forming the basis for the next round of labeling and training. The encoder supports self-supervised pre-training iterative updates, and the labeled model is used for pre-labeled inference.

[0033] The key points of this application mainly include the following two items:

[0034] First, a risk diversion mechanism constrained by semantic consistency and prediction confidence is proposed. This mechanism first identifies samples whose labels may deviate from the local semantic center in the shared semantic space, and then combines high and low thresholds or dynamic calibration thresholds to determine automatic confirmation, careful retention or expert review, thereby reducing the risk of erroneous labels being mistakenly written into the training set and continuing to spread.

[0035] Second, a closed-loop label governance mechanism for network datasets is proposed, which unifies data access, automatic pre-labeling, semantic consistency noise detection, expert review, dataset accumulation and labeling model iteration into a continuous process, so that the output of the labeling model not only serves as a one-time classification result, but also becomes the input for the next round of label governance and dataset update.

[0036] Through the above technical solutions, this application can reduce the reliance on large-scale manual annotation by utilizing weakly supervised, semi-supervised, and self-supervised representation learning, enabling a small amount of expert knowledge or a small number of high-quality strongly labeled samples to be expanded into a continuously iterative network dataset; it can control the label writing boundary through three types of evidence: cluster semantic consistency, prediction confidence, and expert review, reducing the probability of historical noisy labels propagating in subsequent training; it can save the sample source, pre-labeled temporary labels, final confirmed labels, label status, prediction confidence, expert notes, dataset version, and annotation model version, making the label generation process traceable, auditable, and reproducible; and it can also continuously update the network dataset with changes in business protocols, attack patterns, and collection scenarios through dataset accumulation and model iteration write-back, improving the timeliness and scenario adaptability of the dataset. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0038] Figure 1 A schematic diagram illustrating the overall flow of the sustainable labeled network dataset construction method according to an embodiment of this application is shown;

[0039] Figure 2 This paper illustrates a flowchart of data access and standardization processing according to an embodiment of this application.

[0040] Figure 3 A schematic diagram illustrating the shared semantic representation and self-supervised pre-training process of an embodiment of this application is shown;

[0041] Figure 4 A schematic diagram of the process of hybrid supervised pre-labeling and clustering noise detection in an embodiment of this application is shown;

[0042] Figure 5 This illustration shows a flowchart of the prediction confidence level triage and expert review process according to an embodiment of this application.

[0043] Figure 6 This application illustrates a flowchart of the data set accumulation, model iteration, and traceability recording process in an embodiment of this application; and

[0044] Figure 7 A schematic diagram illustrating the system deployment and data security boundary of an embodiment of this application is shown. Detailed Implementation

[0045] To address the problems of high cost of labeling network datasets, easy propagation of noisy labels, difficulty in continuously updating datasets, and lack of traceability in the labeling process, this application proposes a method and system for constructing sustainable labeled network datasets based on hybrid supervised learning. The following description, in conjunction with the accompanying drawings, further illustrates this application. It should be understood that the following embodiments are used to explain the technical solutions of this application and are not intended to limit the scope of protection to a specific model, threshold, data format, or software framework. Those skilled in the art can make equivalent substitutions for specific models, thresholds, storage systems, and user interfaces based on network scenarios, data scales, protocol types, computing power conditions, and review requirements. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0046] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be implemented without employing one or more of the specific details described below, or by using other methods, components, apparatuses, steps, etc. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0047] This application discloses a method for constructing sustainable labeled network datasets based on hybrid supervised learning.

[0048] Figure 1 This diagram illustrates the overall flow of a sustainable labeled network dataset construction method based on hybrid supervised learning, as described in an embodiment of this application. Figure 1 As shown, this process takes continuously accessed network data as the processing object, and relies on the linkage mechanism of sample-level label governance records and data source ledger to build a closed-loop label governance system that is fully covered, traceable, and iterative. It sequentially executes steps such as network data reception, feature processing, semantic representation, hybrid supervised pre-labeling, risk diversion, expert review, and dataset version writing back.

[0049] Step 101: Receive raw network data and its source information. When building a network dataset from scratch, the first step is to receive raw network data, which is then processed to generate samples. In this application, samples are analysis units formed by data streams, sessions, time windows, host behavior, or alarm events. A sample consists of multiple records arranged in chronological order, each record corresponding to a time step during modeling. These time-series records can correspond to feature information of data packets, stream records, log events, or alarm events. For example, a network data stream can be considered as a sample, where each data packet corresponds to feature information of a time step.

[0050] Raw network data can include, but is not limited to, PCAP (packet capture) files, flow records, logs, alarm records, or raw network data obtained from public datasets. Source information may include collection time, collection location, authorization identifier, file hash, data scenario, original tag description, and data usage scope description.

[0051] For example, Wireshark packet capture software can be used to monitor network cards, capture complete packets within a certain time period, and save them as .pcap or .pcapng files as raw network data; based on the five-tuple (source / destination address, source / destination port, and transport protocol), packets belonging to the same network connection within a certain time period can be aggregated to form flow records, without storing the complete content of the packets; the Zeek open-source traffic analysis engine can be used to parse real-time network links or offline PCAP files, outputting multiple structured and categorized log files (e.g., conn.log file contains all TCP / UDP connection records, which can be used to detect abnormal connections; http.log file contains HTTP protocol request and response, which can be used to detect web-type attacks; dns.log file contains DNS interaction information, which can be used to detect malicious domain name resolution; ssl.log file contains TLS (Transport Layer Security) / SSL (Secure Sockets Layer) handshake information, which can be used to detect abnormal certificates and analyze encrypted traffic; files.log file contains records related to file transfer protocols, which can be used to detect malicious file downloads); firewall alarm records can be merged with other network data as raw network data.

[0052] In addition, it is necessary to obtain the source information corresponding to the original network data (i.e., original network data source information). To ensure the tag generation process is traceable, auditable, and reproducible, the original network data source information can be stored in a data source ledger. When dividing the training set, validation set, and test set, the original network data source information can serve as one of the criteria for division. For example, the training set should ideally cover samples from different collection times and locations. Samples from different sources often have different levels of credibility and compliance. The credibility and compliance status of the source can be assessed based on the integrity of the collection chain, authorization credentials, signature / hash, provider reputation, and audit results.

[0053] Among them, the collection time can represent the actual time when the original network data was collected, recorded according to a preset time base and time precision (e.g., milliseconds); the collection location can represent the actual location where the original network data was collected, such as a computer or network transmission device in the network; the authorization identifier can represent a legitimate authorization credential used for compliance auditing; the file hash can represent the hash digest of the file storing the original network data, usually using SHA256 or a stronger algorithm as an integrity verification example, which can be used to verify whether the original network data has been tampered with; the data scenario can represent the business or network environment scenario, such as an internal network with a single gateway; the original label description can represent the initial label information that comes with the public dataset; and the data usage scope description can represent the usage boundaries of the original network data.

[0054] This method also supports continuously flowing multi-source network data. For example, after processing a batch of network data to generate a version of the dataset, the encoder, annotation model, and threshold can be updated; when subsequent batches arrive, the updated configuration can be used to continue building a new version of the network dataset.

[0055] Step 102 involves preprocessing and feature extraction of the raw network data. After segmenting, deduplicating, and anonymizing the fields of the raw network data, it can be aggregated into a network data stream based on 5-tuples, bidirectional sessions, time windows, hosts, or alarm events, and features such as packet length, protocol, address, port, packet time interval, control flags, certificate attributes, domain name statistics, and stream duration can be extracted.

[0056] A single PCAP file may contain numerous network interactions (too many time-series records). During preprocessing, the entire data stream can be segmented, and network data within specific time periods can be filtered and retained to prevent the generated network data stream from becoming excessively long (e.g., no more than 100 time steps). During network data collection, log files may generate duplicate records; deduplication can be performed during preprocessing. Network data may contain sensitive information, such as explicit source / destination addresses and user identifiers; therefore, anonymization can be performed during the preprocessing step. After preprocessing, the preprocessed raw network data can be aggregated into a network data stream, and features of each time-series record (e.g., network data packet) in the network data stream can be extracted. One or more of the following feature types can be extracted as needed.

[0057] Among these, packet length can represent the total length or payload length of a single network data packet. In some cases, the total length or payload length of the data packet is fixed. Protocol can represent the network transport layer and application layer protocol type corresponding to the data packet, such as HTTP (Hypertext Transfer Protocol) and FTP (File Transfer Protocol) at the application layer, and TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) at the transport layer. Address includes source and destination addresses. For IPv4 addresses, they can be split into four integer segments separated by a decimal point. After normalization, each segment is integrated into a set of features, such as source address segmentation features based on the source address. Port includes source and destination ports; commonly used ports include port 80 for the HTTP protocol. Packet time interval is typically used to represent the time difference between adjacent data packets of the same type within the same network connection. For example, in periodic heartbeat communication scenarios, the time interval between heartbeat data packets is relatively stable. Control flags refer to the TCP header control flags (TCP...). Flags include SYN (establish TCP connection), ACK (acknowledge), FIN (normally close connection), RST (reset), PSH (push data), and URG (mark urgent data). A single data packet can carry multiple control flags simultaneously. For example, in the second message of a three-way handshake, the server's reply to the client sets both the SYN and ACK flags. Certificate attributes can represent certificate-related information during TLS interactions. For example, HTTPS communication carries certificate information, and extracting certificate attributes can be used to identify unreliable certificates. Domain name statistics are statistical features obtained based on DNS access behavior of the same connection and the same host. They can be used to detect DGA (Domain Name Generation Algorithm) attacks, where attackers use algorithms to generate a large number of random DGA domain names to circumvent traditional malicious domain name blacklists. Flow duration is a flow-level statistical feature that can represent the total time span of this network data flow, i.e., the difference between the capture time of the first data packet and the last data packet.

[0058] After feature extraction, the features are standardized. Standardization methods include, but are not limited to, max-min normalization, zero-mean (Z-score) standardization, and one-hot encoding. The standardized features are then aggregated to form a feature sequence corresponding to the sample.

[0059] Step 103: Construct the initial data view and data source ledger. Assign a unique identifier to each sample, and use this as the primary key to construct the initial data view (raw view) and data source ledger. Then record the dataset ownership, analysis task number, role classification, label source, label status, and collection batch.

[0060] The initial data view (raw view) uses the sample's unique identifier as the primary key and stores all preprocessed, extracted, and standardized sample feature sequences, covering both unlabeled samples and samples with their own original labels. The data source ledger also uses the sample's unique identifier as the primary key. Besides recording the original network data source information described in step 101, it also records the dataset ownership, analysis task number, role assignment, collection batch, label source, label status, and final label result corresponding to the sample. These collectively constitute the storage content of the data source ledger, serving the functions of sample tracing, version ownership, and link tracing. It does not require storing all label change details. Here, the "data source ledger" can be understood as a logically traceable data structure, not requiring all information to be concentrated in a single data table. In specific implementations, it can be composed of a dataset table, a task table, a sample table (flow table), and audit logs, etc.

[0061] This step synchronously initializes the sample-level label governance record. This record is bidirectionally linked to the data source ledger through the unique sample identifier. It is specifically used to retain the label change details throughout the entire sample lifecycle, running through all subsequent processing steps. It records process data such as label generation, change, confirmation, retention, and rejection, including original label, process label, final label, label source, confidence level, risk indicator, operation record, version information, etc., to achieve auditable and reproducible label processing.

[0062] The dataset attribution indicates which dataset version the sample belongs to. This application supports maintaining and recording multiple iterative dataset versions, rather than just a single fixed dataset version. Some samples in intermediate states are carefully retained and need to be validated using subsequent dataset versions to determine the correctness of their labels. Therefore, it is necessary to distinguish and record the dataset version attribution corresponding to a sample. The analysis task number can be used to distinguish different sample analysis tasks. Each task performs full-process processing on newly added unlabeled samples and also processes the carefully retained samples from previous tasks. The role classification indicates which role the recorded sample is assigned to in training / validation / testing, thus indicating which category the sample belongs to in the training set, validation set, or test set. The collection batch can be used to identify the collection batch and collection order of the original data to which the sample belongs. The label source indicates the source of the label, such as original built-in labels, automatically pre-labeled labels by the model, or manually labeled labels by experts. The label status indicates the lifecycle status of the label, such as unlabeled, pre-labeled, automatically confirmed with high confidence, confirmed by subsequent verification in intermediate state, confirmed by expert review with low confidence, or pre-labeled invalid.

[0063] Step 104 involves performing unified semantic encoding on the standardized feature sequences to obtain a semantic representation (i.e., a semantic vector) for each sample. This semantic vector can simultaneously serve classification reasoning, clustering noise detection, expert review, and model iteration.

[0064] Taking a scenario with 100 time steps and 7 types of features as an example, the sample feature sequence (size [1, 100, 7]) with temporal structure after standardization in step 103 is input into the encoder, and the corresponding sample's global semantic vector without temporal dimension is output. The encoder can use various neural network models to encode the features and generate multidimensional representations. If the original dimension of the sample feature sequence is low (such as the 7 types mentioned above), it can be mapped to a higher-dimensional dense semantic space, and the hidden dimension is set to an integer multiple of the number of attention heads. In some hardware implementations, a dimension that is convenient for parallel computing can be selected. After the encoder completes the aggregation of all temporal information, the dimension representing the time step in the sample feature sequence is eliminated, and the final output semantic vector no longer carries the temporal length dimension.

[0065] Step 105: Perform hybrid supervised pre-annotation. Hybrid supervision in this step refers to integrating multiple annotation paradigms, such as expert rules, weakly supervised annotation functions, historical model predictions, semi-supervised label propagation, model inference, and multi-view collaborative training, rather than relying on a single annotation method, forming a multi-paradigm complementary hybrid annotation system. For ease of explanation, this process is referred to as "pre-annotation using a hybrid supervised annotation model," which can integrate at least two types of supervision sources.

[0066] Among these, expert rules refer to direct judgment based on explicit decision logic summarized by security experts, without the need for model training; for example, if a single source address initiates more than a threshold number of connections to different ports within a short period, the sample can be labeled as a port scan. Weakly supervised labeling functions, compared to expert rules, may have a certain risk of false positives, but noise can be offset by fusing multiple labeling functions. Historical model prediction refers to using previously trained models to infer about samples, but this method is prone to inheriting historical biases, especially with insufficiently trained models. Semi-supervised label propagation refers to label diffusion based on sample similarity, using a small number of existing labeled samples to propagate labels to unlabeled samples with similar features. Multi-view collaborative training refers to constructing two independent feature combinations for the same batch of samples, with two models making predictions separately, and selecting samples where both models achieve high confidence to generate labels.

[0067] To address the inconsistency in the dimensions of output results from various annotation paradigms, a label aggregation and calibration layer is established. This layer can employ at least one of the following methods: weighted voting fusion, validation set temperature scaling calibration, or generative label fusion model. It uniformly converts the outputs from multiple paradigms into standardized probabilities corresponding to each candidate label, uniformly calibrates the multi-source supervision information, and outputs the category probabilities. After obtaining the probabilities of each candidate label, the highest predicted probability is used as the prediction confidence, and the candidate label corresponding to this highest predicted probability is determined as the temporary label.

[0068] The annotation model in this step can use one annotation paradigm or a combination of multiple annotation paradigms to perform pre-annotation based on the semantic vectors of the samples to generate temporary labels. During pre-annotation, the prediction confidence, annotation rule evidence, and label source information are saved simultaneously to classify samples into low prediction confidence samples, intermediate prediction confidence samples, and high prediction confidence samples.

[0069] The temporary labels, initial prediction confidence levels, labeling rule evidence, and label source information generated in this step are all written to the sample-level label governance record in real time, and the sample label status is updated synchronously to pre-labeled, completing the initial step of the label governance process for tracing and preserving evidence.

[0070] Step 106 involves risk triage of samples based on semantic consistency and prediction confidence to identify risky samples. After obtaining the semantic vectors of the samples, K-Means clustering, spectral clustering, density clustering, similarity retrieval, or other local consistency algorithms can be used for analysis. Based on the clustering relationships in the vector space, samples within the same cluster whose main labels are inconsistent with those within the same cluster are identified.

[0071] The similarity between sample semantic vectors represents the semantic consistency of sample behavior. If multiple sample semantic vectors are similar, it indicates that the traffic behavior characteristics are similar and they can be classified into the same cluster. Based on local semantic distribution characteristics, intra-cluster label consistency can be used as a risk identification clue, but this clue is only used for risk warning and is generally not used as a decisive basis for determining the authenticity of labels. The identification process is recommended to configure multiple constraints: for example, setting a minimum cluster size threshold to filter out excessively small noisy clusters; introducing a neighborhood purity index to assist in evaluating intra-cluster label distribution; performing normalization on semantic distance and adding a boundary sample protection mechanism to reduce the risk of misidentification in boundary samples, homomorphic traffic, and class imbalance scenarios. When the temporary labels of a small number of samples within a cluster conflict with the labels of the majority of samples within the cluster, these samples are marked as candidate risk samples. Simultaneously, combined with the pre-labeled prediction confidence, candidate risk samples with low prediction confidence and intra-cluster label conflicts are given priority for identification as risk samples.

[0072] The semantic anomaly identifiers, tag conflict reasons, and risk level determination results obtained in this step are all recorded in the sample-level tag governance record, providing a complete risk tracing basis for subsequent sample diversion and expert review.

[0073] Step 107 involves categorizing samples into three types: automatic confirmation, cautious retention, and expert review. Samples with high-confidence temporary labels and consistent semantics are automatically added to the dataset (automatic confirmation); samples in intermediate states retain their original temporary labels and await verification in subsequent versions (cautious retention); samples with low confidence or semantic abnormalities are entered into the expert review queue (expert review). Automatic confirmation samples are configured with a single-category sample ratio limit for inclusion in the dataset and participation in model training, along with regular sampling and historical sample replay verification mechanisms to mitigate the accumulation of confirmation bias.

[0074] This step updates the authoritative label status of the sample in real time based on the triage results, and writes the triage type, handling strategy, and reasons for suspension into the sample-level label governance record, realizing full traceability of label risk classification governance and completing the core link of risk control in label governance.

[0075] The samples with temporary labels generated after pre-annotation are divided into the three categories mentioned above and processed separately to complete the update of the network dataset. For expert review, the samples are sent to the expert review queue, the expert confirmation labels are received, the review records are written into the sample's label governance record, and the expert confirmation labels are written into the dataset as confirmed labels.

[0076] Step 108: Write the confirmed labels, review records, model training results, and model version, along with the sample data, to generate a dataset version that can be used for the next round of training, annotation, and review. Specifically, if model training is triggered in this round, the corresponding model training results and new model version information are written; if model training is not triggered in this round, the model version used in this round is recorded, and no new training results are needed.

[0077] Confirmed labels include automatically confirmed labels and expert-confirmed labels obtained through expert review. This step summarizes the governance data throughout the entire process, archiving the final confirmed labels, review records, model versions, dataset versions, and iteration batch information into the sample-level label governance record. At the same time, it updates the final sample status and version snapshot in the data source ledger, completing the single-round label governance closed loop.

[0078] The dataset version updated in step 108 serves as the data foundation for semantic encoding and model training in step 104 of the next process, forming a closed loop of encoding—pre-labeling—clustering noise detection—splitting confirmation—dataset update—model iteration.

[0079] Figure 2The diagram illustrates a data access and standardization process according to an embodiment of this application. This process converts authorized network data into a unified feature sequence that is labelable, trainable, and traceable, providing consistent input for subsequent hybrid supervised annotation.

[0080] Step 201: Receive authorized raw network data. This corresponds to step 101 above. Specifically, for PCAP file data, it can be read packet by packet and aggregated based on 5-tuples or bidirectional sessions; for NetFlow / IPFIX or Zeek logs, flow-level fields can be read directly; for alarm records or device logs, they can be aligned and merged with the aforementioned network data based on event time, host identifier, or host task sequence number to form raw network data with associated alarm and device information. Legally authorized raw network data can be used compliantly to construct network datasets.

[0081] Step 202 involves performing minimal data collection and data anonymization. This corresponds to the preprocessing in step 102 above. Specifically, for the payload content, only the payload length, transmission direction, summary, or protocol metadata can be retained to reduce the storage of redundant data. For sensitive fields such as source / destination addresses, domain names, URLs, certificate attributes, and user identifiers, hashing, generalization, truncation, summarization, or retaining statistical features while deleting the original values ​​can be used to complete data anonymization, ensuring data compliance while meeting the requirements of feature analysis.

[0082] One-way hashing typically cannot reconstruct the original content and is considered an anonymization and desensitization method. Generalization, truncation, and digestion methods reduce information granularity and prevent the leakage of precise entity information. This solution can employ a combination of various desensitization methods.

[0083] Step 203 transforms samples from different sources into a unified feature sequence. This corresponds to feature extraction in step 102 above. Specifically, after feature extraction, the features are standardized, and then the standardized features are aggregated to form a feature sequence corresponding to the samples. In one embodiment, each flow retains a maximum of 100 time steps, and each time step contains 7 dimensions of features: length (packet length), protocol, src_ip_segments (source address segments), dst_ip_segments (destination address segments), src_port (source port), dst_port (destination port), and flags (control flags). In other embodiments, these can also be expanded to include features such as packet time interval, transmission direction, TLS handshake fingerprint (generated from TLS handshake negotiation parameters), domain name statistics, or alarm context.

[0084] Step 204 prioritizes dividing the training, validation, and test sets based on the original network data source information, collection batch, or analysis task number, rather than simply using random partitioning. This corresponds to step 103 mentioned earlier. This process reduces the risk of data leakage caused by samples from the same source simultaneously entering training and testing, and improves the reproducibility of subsequent model and dataset versions.

[0085] This process ultimately outputs standardized feature sequences (generated by step 203), a data source ledger, and sample role assignments (training / validation / testing, generated by step 204) to subsequent steps.

[0086] Regarding the role of sample classification, samples are divided into training set, validation set, and test set according to batch and source. The dataset roles of each type of sample remain unchanged in subsequent dataset version iterations. Among them, test set samples are only used for model performance evaluation and must not participate in model training, threshold fitting, or weight learning of label aggregation calibration layer; validation set samples are only used for hyperparameter selection, probability calibration, and threshold tuning, and are generally not incorporated into the training set for final model training.

[0087] Figure 3 This diagram illustrates the shared semantic representation and self-supervised pre-training process of an embodiment of this application. This process provides a consistent sample representation foundation for classification reasoning, clustering noise detection, and expert review, avoiding conflicting interpretation results caused by different modules using inconsistent features. This part corresponds to step 104 above.

[0088] Step 301 involves inputting the feature sequence of each network data stream or session into the shared encoder. This step corresponds to the unified semantic encoding in step 104 above. Specifically, the encoder uses the same set of network parameters to process all samples, and the feature sequences corresponding to various streams or sessions reuse the same encoder parameters or weights. Using a shared encoder enables the model to learn general temporal representations of multiple types of network data streams and sessions, achieving parameter reuse and helping to improve the model's generalization ability to different types of traffic. The encoder can employ models such as Transformer, LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), TCN (Temporal Convolutional Network), CNN (Convolutional Neural Network), or GNN (Graph Neural Network).

[0089] For a small-scale experiment, a Transformer encoder can be used with an input dimension of 7, a hidden dimension of 128, 4 multi-head attention heads, 2 encoding layers, and a maximum sequence length of 100. A temporally structured sample feature sequence (size [1, 100, 7]) is input into the encoder. The encoder first maps the 7-dimensional original features to a 128-dimensional hidden space and introduces positional encoding to represent temporal relationships. It then uses 4 multi-head attention heads to mine the dependencies between time steps, and performs feature transformation and temporal information fusion layer by layer through 2 encoding layers. Finally, global aggregation eliminates the 100-dimensional time step dimension, outputting a global semantic vector of size [1, 128].

[0090] In other embodiments, the encoder structure can also adopt various model combinations. For example, protocol behavior features are often related to the source port, destination port, and local patterns of the message sequence. Therefore, a one-dimensional CNN can be added to extract local correlation features of the message within a short window in parallel. The output features of the CNN can be concatenated or weighted and fused with the output features of the temporal encoder to enrich the feature representation.

[0091] The shared encoder performs two core processes: mapping the original features to a higher-dimensional space, mining temporal correlations, fusing all temporal information, removing the temporal dimension of the sequence, and outputting a global representation that does not contain temporal length information.

[0092] Step 302, optionally, involves performing masked autoencoder self-supervised pre-training. Specifically, features from a portion of the time steps in the feature sequence of the sample are randomly masked according to a preset mask ratio. Features at the masked locations are reconstructed using the unmasked context. Reconstruction loss is calculated only at the masked locations, and the encoder is pre-trained self-supervised based on this mask reconstruction task. This process can utilize unlabeled or weakly labeled samples to learn basic temporal representations, and can still initiate the representation learning process and complete the shared encoder weight update even in scenarios where strong labels are scarce.

[0093] The updated shared encoder is obtained through pre-training with mask autoencoder in step 302, which can be used to perform sample semantic embedding in step 301 to generate sample semantic vectors. The pre-training iteration brings about encoder version upgrades, thereby updating the representation results of all samples, which may cause changes in classification inference results and clustering partitioning results.

[0094] The encoder does not need to be updated in every processing round. The encoder can be self-supervised trained using newly added unlabeled or weakly labeled samples, or it can be fine-tuned during supervised training.

[0095] Step 303 involves pushing the semantic vector downstream. Semantic vectors output from the same shared encoder can be used simultaneously for classification inference, clustering noise detection, and expert review. Because these tasks share the same semantic space, the suspicious causes, sample clustering relationships, and model predictions seen by experts have a consistent representational basis, avoiding inconsistencies in interpretation between different analytical results.

[0096] If classification reasoning and clustering noise detection use two independent encoders, the same sample will generate two sets of semantic vectors through different encoders. Classification reasoning outputs a decision based on one semantic vector, while clustering noise detection obtains a decision based on the other semantic vector. The two semantic spaces are inconsistent, which can easily lead to contradictory decision conclusions. This problem becomes more prominent when the encoder is not sufficiently trained.

[0097] Figure 4 A schematic diagram of the hybrid supervised pre-labeling and clustering noise detection process according to an embodiment of this application is shown. This process is used to generate temporary labels based on multi-source supervision information and identify label risks in a shared semantic space. This part corresponds to step 105 above.

[0098] Step 401: Generate temporary labels based on available supervisory information. Supervisory information may include direct expert annotations, port or protocol rules, file path or directory name hints, threat intelligence, sandbox execution results, historical model predictions, labels from the same dataset, and weakly supervised annotation functions. This information comes from different acquisition channels, constituting multi-source supervisory information. Each label records its source type and evidence in the data source ledger for easy auditing and tracing.

[0099] Step 402 involves clustering or neighborhood retrieval of samples within the shared semantic space. The shared semantic space is a unified representation space mapped by a unified shared encoder, where all samples are calculated based on the same semantic benchmark to ensure consistency in feature representation. This step uses unsupervised clustering and neighborhood retrieval to uncover implicit similarity relationships between samples. Each sample sub-cluster formed by clustering represents a local semantic structure, and samples within the same local semantic structure possess highly similar semantic features and should have consistent label attributes.

[0100] This step relies on the constructed local semantic structure to verify whether there are conflicts, outliers, or inconsistencies in the temporary labels generated by the aforementioned multi-source supervision, thereby determining the semantic consistency of the labels of a single sample.

[0101] This embodiment uses K-Means as an example. The number of clusters can be set to a multiple of the number of categories, rather than simply being the same as the number of categories. This supports the local semantic structure of multiple sub-clusters corresponding to the same semantic label, thus preserving the fine-grained sub-pattern differences within the same semantic label. For example, the malicious traffic category can include subcategories with differences, such as scanning short connections and long-term Trojan communication, while the normal traffic category can include different business subcategories such as video transmission, office access, and file transmission. Each subcategory corresponds to an independent local semantic structure, thereby achieving refined semantic distribution modeling and providing an accurate semantic benchmark for subsequent semantic consistency judgment and label risk diversion.

[0102] Step 403: Statistically analyze the label distribution within each cluster and identify samples that are significantly inconsistent with the main label, neighborhood label, or historical label within the cluster. These samples are then designated as semantically anomalous samples. The main label within a cluster refers to the temporary label carried by most samples within the same sub-cluster. Neighborhood labels refer to the labels carried by a group of samples that are closest to the current sample in the shared semantic space and have similar semantic vectors. Similar semantic vectors indicate similar sample behavior patterns; if the labels of surrounding samples differ from the current sample's own label, a conflict arises between the sample representation and the label information. Historical labels refer to the long-term labeled labels corresponding to the same network entity (e.g., the same computer or network transmission device) in historical data.

[0103] This step does not directly take the majority label within the cluster as the objective truth value. Instead, it marks samples that deviate from the local semantic center as noise candidates, leaving subsequent steps to determine whether to automatically confirm or have them reviewed by experts. The advantage of this approach is that it avoids simply using a "majority rule" mechanism to forcibly cover suspicious samples, preventing the further spread of batch erroneous labels. At the same time, it does not directly determine whether the labels are right or wrong, but handles suspicious samples flexibly, balancing the efficiency of automatic processing with the reliability of expert review through flexible triage.

[0104] A sample can be identified as semantically anomalous by using semantic anomaly identifiers. These identifiers are part of the label risk information for a single sample, are associated with the sample's unique identifier, and are stored in the sample record or label governance record.

[0105] Step 404: Calculate the risk score by combining the risk score evaluation indicators. These indicators may include entropy, maximum prediction probability, vector distance from the sample to its cluster center, neighborhood label consistency rate, number of label function conflicts, sample source credibility, and historical verification results.

[0106] Among them, entropy represents the dispersion of the predicted probabilities of each category; the higher the entropy, the more difficult it is to determine the sample's classification. The maximum predicted probability is the highest value among the predicted probabilities of each category. This value reflects the credibility of the current judgment; the higher the value, the more credible the current label judgment; the lower the value, the more likely the judgment is biased. The vector distance from the sample to its cluster center refers to the distance from the sample vector in the shared semantic space to the center vector of its sub-cluster. The greater the distance, the greater the difference in behavior between the sample and the mainstream samples within the sub-cluster. The neighborhood label consistency rate is the proportion of neighboring samples in the semantic space whose labels are the same as the current sample's label; the lower the consistency rate, the more obvious the label conflict phenomenon in the local area. The label function conflict number is the number of rules whose judgment conclusions contradict each other among the multiple labeling rules that generate temporary labels. The sample source credibility is classified into reliability levels according to the acquisition channel of the temporary label, used to measure the credibility of the original basis of the label. For example, temporary labels obtained based on expert rules are relatively credible. Historical review results record whether the sample has been reviewed by experts in the past and whether there are any label adjustment records.

[0107] For example, the risk score evaluation indicators can be standardized and the final risk score calculated as follows: All evaluation indicators are normalized to the same dimension; the maximum prediction probability, neighborhood label consistency rate, and sample source credibility are converted into inverse risk dimensions. Higher original values ​​for these indicators represent lower risk, while higher values ​​after inverse conversion represent higher risk; after normalization and inverse conversion, all evaluation indicator results are weighted and summed according to preset weights to output the final risk score. The weights for each indicator can be configured as fixed weights, weights obtained based on validation set performance calibration, or weights manually configured according to the business scenario.

[0108] Based on the risk score, noisy candidate samples are ranked, and an expert review queue is constructed. The queue can be dynamically rearranged according to the risk score, so that the limited expert review resources can prioritize the processing of samples that have a greater impact on the quality of the dataset.

[0109] Figure 5 A flowchart illustrating the prediction confidence level triage and expert review process according to an embodiment of this application is shown. As shown, this process determines whether a sample enters the automatic confirmation, cautious retention, or expert review path based on the prediction confidence level, semantic anomaly identifier, and risk score. This part corresponds to steps 106 to 108 above.

[0110] Step 501: Dynamically calibrate the thresholds for the three processing paths based on model version, sample category distribution, expert workload, and business risk.

[0111] Samples are divided into intervals based on prediction confidence: samples with confidence levels above the automatic confirmation high confidence threshold are automatically confirmed; samples with confidence levels below the expert review low confidence threshold are subject to expert review; samples between the two thresholds are cautiously retained. Traffic splitting can be adjusted based on factors such as model version, sample category distribution, expert workload, and business risks. The model version reflects the current model inference accuracy. When model accuracy declines, the automatic confirmation high confidence threshold should be tightened by increasing this threshold to reduce the passage of erroneous labels. Conversely, as model performance improves, this threshold can be appropriately relaxed to increase the proportion of automated processing. Sample category distribution reflects the balance of different traffic sample types. When the number of samples in each category is unbalanced, or the proportion of newly added attack-type samples suddenly increases, the expert review low confidence threshold can be appropriately increased to include more niche samples for expert review. Manual annotation can supplement the labels of niche categories, continuously iterating the model to improve the prediction confidence distribution of niche samples, preventing niche category samples from being concentrated in the review queue due to consistently low confidence levels, thus balancing the processing strategy for different types of samples. Expert workload can be used to balance the expert review load. When there are sufficient expert review personnel, the low confidence threshold for expert review can be raised, allowing more questionable samples to be manually verified. Conversely, when expert resources are scarce, the threshold can be lowered to compress the review range and expand the range of automatically confirmed samples, alleviating the pressure on manual processing. For business risks, the high confidence threshold for automatic confirmation can be tightened based on network security level control label access standards. This raises the confidence threshold for automatic confirmation in high-risk network scenarios (such as core business intranets and attack and defense detection scenarios), strictly controlling the inflow of noisy labels into the dataset. For ordinary office scenarios, the threshold can be appropriately relaxed to balance labeling efficiency.

[0112] In one embodiment, the maximum predicted probability can be used as an indicator of prediction confidence. When the maximum predicted probability of a noisy candidate sample is greater than or equal to 0.9, it is in the high prediction confidence interval; when the maximum predicted probability is less than or equal to 0.7, it is in the low prediction confidence interval; and when the prediction confidence is in between, it is in the intermediate prediction confidence interval. The automatic confirmation of the high confidence threshold and the expert review of the low confidence threshold can be dynamically adjusted according to business risks and model versions.

[0113] Step 502: Based on the prediction confidence and semantic anomaly identifier, the samples are routed to different processing paths. If a sample is routed to expert review, the samples in the expert review queue are then sorted according to the risk score. The prediction confidence can be indicated by the maximum prediction probability. The semantic anomaly identifier can indicate low semantic consistency. The risk score is the risk score calculated in step 404 in conjunction with the risk score evaluation index.

[0114] In one embodiment, triage is performed based on prediction confidence and risk conditions: For samples with high prediction confidence and semantic consistency, a temporary label is used as the automatic confirmation result, and this temporary label is stored as a confirmed label in the clean view; for samples with intermediate prediction confidence or insufficient risk evidence, the temporary label is not rewritten, and they are included in the candidate set to be verified or the data view to be verified as cautiously reserved samples, and re-evaluated after subsequent model version updates, threshold calibration, or expert sampling; for samples with low prediction confidence, semantic anomalies, label function conflicts, or inconsistent neighborhood labels, the label is not directly rewritten, but is pushed to the expert review queue, and the risk cause, prediction confidence, original label, temporary label, and relevant evidence are retained; for samples that have been reviewed and confirmed by experts, the expert judgment result is updated to a confirmed label, and the operator, operation time, original label, temporary label, confirmed label, prediction confidence, risk cause, and remarks are recorded.

[0115] Step 503: During expert review, only samples with low confidence, semantic anomalies, label conflicts, or high impact are accepted, instead of having experts process all samples. Experts can confirm model labels, rewrite labels, skip samples, add notes, or process similar samples in batches. The operator, operation time, original label, temporary label, confirmed label, prediction confidence, risk reasons, and notes can be saved in the sample-level label governance record.

[0116] Step 504: Confirm the results and write them back to the sample table and dataset version table. For automatically confirmed samples, save the automatically labeled evidence; for expert-reviewed samples, save the expert-reviewed evidence; for samples that are temporarily uncertain, the system retains them in a pending verification state and can re-evaluate them after subsequent model version updates.

[0117] The sample table (i.e., the flow table) can include the following fields: current valid label (label), pre-labeled temporary label (temp_label), confirmed label (true_label), and sample status (status, e.g., pending review, confirmed by experts). The dataset version table can record different versions of the dataset.

[0118] Figure 6 The diagram illustrates a workflow for dataset accumulation, model iteration, and traceability recording according to an embodiment of this application. As shown, this workflow is used to write confirmation tags, data views, model versions, and training metrics into a traceable dataset management structure.

[0119] Step 601: Organize the samples into multiple data views according to their label governance status. The label governance status is stored in the sample-level label governance record. The raw view stores all original samples after standardized feature extraction, including unlabeled samples and original labeled samples, used as a baseline or for tracing the source of original data. The clean view retains high-confidence samples filtered for semantic consistency noise. The refined view adds samples evaluated in subsequent versions or confirmed by experts to the clean view, serving as the final dataset for formal model training. Automatically confirmed samples must simultaneously meet the requirements of high confidence, semantic label consistency, and passing quality sampling. A maximum percentage of each type of automatically confirmed sample is set, and a mechanism of regular sampling and historical sample playback is used to avoid the continuous accumulation of confirmation bias.

[0120] Step 602: Save the dataset version, model version, and training metrics.

[0121] In one embodiment, the model table stores fields such as version, accuracy (the percentage of correctly predicted samples out of all samples), precision (the percentage of true positive samples among those predicted as positive samples), recall (the percentage of true positive samples identified), F1 score (the harmonic mean of precision and recall), training_data_size (the size of the training dataset), training_duration (the training duration), parameters (the number of model parameters), and architecture (the model network architecture). The model can be recorded according to different versions, rather than retaining only one version. The flow table stores fields such as label (the final label of the sample, i.e., the confirmed label), temp_label (the temporary label of the sample), true_label (the true label of the sample), confidence (the prediction confidence), status (the sample processing status), features (the sample feature vector), dataset_id (the dataset ID), analysis_task_id (the analysis task ID), and notes (notes). A sample can be associated with multiple label fields. The above fields can be replaced or extended depending on the deployment environment.

[0122] The annotation model does not need to be updated in every processing round. The annotation model used to generate temporary labels mainly utilizes expert-confirmed samples, as well as automatically confirmed samples that simultaneously meet the requirements of high prediction confidence, consistency between labels and local semantics, and passing quality sampling, and other reliable samples for supervised or semi-supervised updates. An upper limit is set on the proportion of each class in the automatically confirmed samples participating in training, and sample quality sampling and historical sample replay verification are performed periodically to avoid the cumulative amplification of confirmation bias caused by continuous iteration of automatically confirmed samples of a single class. Training and the generation of a new version are only triggered when the accumulation of new reliable samples reaches a specified number, the periodic training conditions are met, or the model performance deteriorates. In rounds where training is not triggered, the annotation model maintains fixed weights and only performs sample inference.

[0123] Step 603: After model training is complete, the model records are written back to the online annotation service. The new model does not directly overwrite the historical model; instead, it stores data for each version of the model based on the version field to support model rollback operations. When new samples enter the system subsequently, the current online model can be used for pre-annotation, or a specific model version can be selected to replay the annotation process as needed to complete the next round of dataset updates.

[0124] Step 604 generates a label governance report, including data source distribution, automatic labeling ratio, expert review ratio, number of noise candidates, number of corrections, training effect of different data views, threshold sensitivity results, and list of abnormal samples, for operation and maintenance personnel, data administrators or security experts to review, and supports the reproduction of the label governance process.

[0125] The data source distribution is used to statistically analyze the proportion of samples from different collection channels, batches, and original data types, assessing the diversity of dataset sources. The automatic labeling ratio refers to the proportion of samples automatically confirmed with high confidence and requiring no manual intervention, reflecting the efficiency of automated labeling. The expert review ratio refers to the proportion of samples that enter the expert review queue and undergo manual processing, reflecting the scale of manual input. The number of noisy candidates represents the total number of suspicious samples identified by clustering semantic consistency detection as having label conflicts or semantic deviations. The number of corrections represents the number of samples whose original temporary labels were modified or corrected during expert review, reflecting the number of samples that were manually corrected. The training effect of different data views refers to the precision, recall, and other metrics obtained by training the model based on the original sample view (raw view), the denoised clean sample view (clean view), and the expert-refined sample view (refined view), used to compare the model performance differences under different data views. The threshold sensitivity result represents the change in the number of automatically confirmed and reviewed samples under multiple sets of confidence and semantic consistency thresholds, serving as a reference for subsequent dynamic parameter tuning. The list of anomalous samples summarizes all suspicious samples with low confidence, semantic outliers, label conflicts, and high-risk scores, facilitating centralized verification of dataset defects by experts.

[0126] Figure 7 A schematic diagram illustrating the system deployment and data security boundary of an embodiment of this application is shown. As shown, this deployment method is used to illustrate the data processing boundary between the acquisition terminal 701, the annotation server 702, the training server 703, the expert terminal 704, the storage device 705, and the audit log 706.

[0127] The technical solution of this application can be deployed as a standalone research platform, an intranet annotation server, a collaborative system of edge nodes and a central training server, or an annotation service integrated with SIEM (Security Information and Event Management) / SOAR (Security Orchestration Automation and Response), IDS (Intrusion Detection System) / IPS (Intrusion Prevention System), probes, gateways, and data lakes. The data acquisition end only uploads authorized data; the annotation server performs parsing, inference, queue scheduling, and dataset management; the expert terminal only accesses the samples to be reviewed and the necessary context; the audit log records all label changes.

[0128] To ensure data compliance, the system can be configured with access control, sample anonymization, malicious sample isolation, model version rollback, log retention, and sensitive field minimization policies. Data involving attack traffic is used only for detection, defense, forensics, evaluation, or simulation, and does not output attack execution instructions or steps to circumvent security mechanisms.

[0129] Based on the above modules and steps, this application solves the problems of high annotation cost, easy accumulation of noisy labels, difficulty in focusing review resources, and difficulty in long-term maintenance of network datasets during continuous construction by using hybrid supervised pre-annotation, shared semantic representation, semantic consistency noise detection, prediction confidence splitting, expert review, and versioned write-back. This forms a sustainable, traceable, and iterative method for constructing network datasets.

[0130] This application also discloses a system for constructing sustainable labeled network datasets based on hybrid supervised learning. This system may include the following modules.

[0131] (1) Data access and standardization module: Receives raw network data, processes the raw network data to obtain standardized feature sequences.

[0132] Raw network data can include, but is not limited to, PCAP files, flow records, logs, and alarm records. It can also support importing public datasets and authorized traffic data collection. Then, the raw network data is processed by flow / session segmentation, sample deduplication, field anonymization, feature encoding, and time-series window partitioning to generate standardized feature sequences. At the same time, the data collection source is registered and the training / validation / testing roles are assigned.

[0133] (2) Semantic representation module: The encoder is used to perform unified semantic encoding on the standardized feature sequence to obtain the semantic vector corresponding to each sample.

[0134] This module receives samples carrying standardized feature sequences, which can be divided into two categories: unlabeled samples and weakly labeled samples. The semantic representation module contains a shared encoder. This encoder uses the same set of network parameters to process all samples, and the feature sequences corresponding to various streams or sessions reuse the same encoder parameters or weights. Before encoding, a Transformer-based sequence encoder can be pre-trained with masking self-supervision. Then, this encoder generates stream-level or session-level semantic vectors, outputting the semantic vectors (also known as sample embeddings) corresponding to each sample, the encoder version, and a unified shared semantic representation (i.e., a shared semantic space). All sample embeddings are mapped to the common vector space corresponding to this shared semantic representation.

[0135] (3) Pre-labeling module: Based on semantic vectors, the labeling model is used to perform hybrid supervised pre-labeling on the samples to generate temporary labels and their prediction confidence for the samples.

[0136] This module receives shared semantic representations, expert rules, a small number of strong labels, existing weak labels, and historical models. It integrates multiple hybrid supervision methods, such as labeling functions, rule voting, semi-supervised label propagation, model inference, or collaborative training, to generate temporary labels and initial prediction confidence.

[0137] The annotation model integrates multiple annotation paradigms, including expert rules, weakly supervised annotation functions, historical model prediction, semi-supervised label propagation, model inference, and multi-view collaborative training, moving away from reliance on a single annotation method and forming a multi-paradigm complementary hybrid annotation system. This step can use one or a combination of annotation paradigms to pre-annotate samples based on their semantic vectors to generate temporary labels. Strong labels typically represent annotation results that have been verified by experts and have the highest credibility, while weak labels are usually annotation results generated based on rule / model inference and have not undergone manual verification. This module enables pre-annotation of samples with only a small number of high-quality strongly labeled samples, a small number of expert rules, or existing weak labels, for use in subsequent steps such as noise detection and confidence splitting.

[0138] (4) Noise detection and confidence-based diversion module: Based on semantic consistency and prediction confidence, risk samples are identified and the samples are diverted.

[0139] This module receives temporary labels, shared semantic representations, historical labels, and prediction confidence (e.g., maximum prediction probability). Based on local semantic consistency, label distribution within clusters, entropy values, maximum prediction probability, and threshold strategies, it identifies high-risk samples and performs three-way splitting according to their confidence level, distinguishing between automatically confirmed, cautiously retained, and expert-reviewed samples, thereby outputting automatically confirmed samples, cautiously retained samples, and samples awaiting review.

[0140] (5) Triage module: Differentiate the processing of various samples according to the triage results, automatically generate automatic confirmation labels, receive expert operation output expert confirmation labels, retain temporary labels of carefully reserved samples for subsequent review, and write the confirmed labels back to the samples. The confirmed labels include automatic confirmation labels and expert confirmation labels.

[0141] The module performs expert review on the samples to be reviewed and writes back the expert confirmation labels. It receives the samples to be reviewed, the reasons for the risks, the sample characteristics, and historical records, pushes them to the expert queue according to priority, receives manual confirmation, modification, skipping, and remarks, and retains the operator, time, and evidence. It outputs expert confirmation labels, review records, and audit logs.

[0142] Carefully retain samples for re-evaluation after subsequent model version updates, threshold calibrations, changes in reference samples, or expert sampling. If a new version of the encoder or annotation model has been developed in subsequent rounds, the semantic vector of the sample should be regenerated using the new encoder, the temporary label and predicted probability should be regenerated using the new annotation model, the prediction confidence should be recalculated based on the new prediction probability, clustering or neighborhood semantic consistency should be re-performed in the updated semantic space, and finally, the sample should be re-triggered based on the new prediction confidence, semantic consistency, and risk score. If a new version of the encoder or annotation model has not yet been developed in subsequent rounds, only the necessary items should be recalculated, i.e., only the metrics affected by the update should be recalculated. For example, if only the triage threshold is adjusted, the original predicted probabilities can be reused, and only the threshold triage needs to be re-executed; if the encoder remains unchanged, but the reference sample set or clustering range changes, the semantic vectors can be reused, but the clustering should be re-clustered or the neighborhood semantic consistency should be recalculated; if the labeling model is updated, the temporary labels and prediction confidence must be recalculated; if the encoder is updated, the semantic vectors must be regenerated and the semantic consistency must be re-evaluated; if the model, threshold, and reference samples have not changed, repeated calculations usually do not produce new results, and there is no need to repeat the processing steps.

[0143] (6) Data accumulation and iterative update module: This module receives automatically confirmed samples, expert confirmed samples, model training results and version indicators, divides them into multiple data views (raw / clean / refined) according to the label credibility status, persistently stores the dataset version, encoder version, training indicators and reproducible sample partitioning scheme, and outputs continuously updated network datasets, model records and label governance reports.

[0144] This module is responsible for the closed-loop label governance across the entire chain, managing the entire lifecycle of samples from initial candidate labeling, risk screening, manual verification, label status updates, result accumulation, version retention to iterative backflow; it uniformly maintains label sources, label status, confidence information, review records, dataset versions and model versions, enabling labels to be traceable, auditable, and rollbackable; it drives the continuous iterative updates of datasets and models, forming a reusable and self-optimizing closed-loop label governance system.

[0145] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium. As is known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media generally include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. Computer program products are software products that primarily implement their solutions through computer programs. The computer program code or computer instructions they contain can be obtained from computer-readable storage media and can also be transmitted, distributed, and downloaded over a network.

[0146] Although the embodiments of this application are described above, the content is merely an example adopted for the purpose of facilitating understanding of this application and is not intended to limit the scope and application scenarios of this application. Any person skilled in the art described in this application may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be determined by the scope defined in the appended claims.

Claims

1. A method for constructing a network dataset, characterized in that, The construction method includes: Receive raw network data, process the raw network data to obtain standardized feature sequences; An encoder is used to perform unified semantic encoding on the standardized feature sequence to obtain the semantic vector corresponding to each sample. Based on semantic vectors, a labeling model is used to pre-label samples, generating temporary labels for the samples and their predicted confidence scores. Based on semantic vectors, a local semantic structure is constructed through clustering. Clustering noise detection is then performed on pre-labeled samples to determine semantic consistency, where semantic consistency characterizes the degree of consistency between temporary labels and the local semantic structure; and Risk triage is performed on pre-labeled samples by combining prediction confidence and semantic consistency, and different types of samples are handled differently according to the triage results.

2. The construction method according to claim 1, characterized in that, The construction method also includes retaining full-process tag governance records, which include: When receiving raw network data, the source information of the raw network data is collected synchronously. After extracting standardized feature sequences, a data source ledger is constructed, and a label governance record is initialized for each sample. When performing pre-labeling on samples, the pre-labeling information is written into the label management record of the corresponding sample; When performing clustering noise detection on samples, the noise detection information is written into the label management record of the corresponding sample; When performing sample splitting, the sample splitting information is written to the corresponding sample's label management record; and When handling different types of samples in a differentiated manner, the handling results information will be written into the sample's label management record; The pre-labeling information includes at least temporary labels and predicted confidence levels; the noise detection information includes at least the reasons for label conflicts; the sample diversion information includes at least the diversion type; the handling result information includes at least the confirmed labels; and the data source ledger and label governance records are linked through the unique sample identifier.

3. The construction method according to claim 1 or 2, characterized in that, Based on prediction confidence and semantic consistency, pre-labeled samples are split into groups, and different types of samples are processed differently according to the splitting results, including: Obtain the probability output by the labeling model for each candidate label, and take the highest predicted probability as the prediction confidence. The K-Means clustering algorithm is used to cluster the semantic vectors of samples to form local semantic structures. The distance from the semantic vector of a sample to the semantic vector of the center of its cluster is calculated to represent semantic consistency. The temporary labels of samples within the same cluster are compared to identify samples with conflicting labels within the cluster. Based on the first set of thresholds, the prediction confidence of the samples is divided into three categories: high prediction confidence, intermediate prediction confidence, or low prediction confidence. Based on the second set of thresholds, the samples are divided according to their semantic consistency, resulting in two categories: semantically consistent or semantically abnormal. Based on prediction confidence and semantic consistency, pre-labeled samples are categorized into three groups for differentiated processing: automatically confirmed, cautiously retained, and expert-reviewed; and Generate a continuously updated network dataset based on the differentiated samples.

4. The construction method according to claim 3, characterized in that, When a sample has high prediction confidence and the temporary label is semantically consistent with the local semantic structure, the sample is delineated for automatic confirmation; when a sample has low prediction confidence or the temporary label is semantically abnormal with respect to the local semantic structure, the sample is delineated for expert review; when a sample does not fall into either of the above two categories, the sample is delineated for cautious retention.

5. The construction method according to claim 3, characterized in that, When a sample is split into automatic confirmations, the temporary label corresponding to the sample becomes the automatic confirmation label, and the automatic confirmation label is written into the dataset as a confirmed label. When a sample is assigned to an expert review panel, it is sent to the expert review queue. The panel receives expert confirmation labels, writes the review record to the sample's label governance record, and adds the expert confirmation label as a confirmed label to the dataset. When a sample is diverted to a cautious retention setting, the corresponding temporary label is retained to await verification in subsequent versions.

6. The construction method according to claim 5, characterized in that, The samples in the expert review queue are sorted based on risk scores, wherein the risk scores are calculated based on risk score evaluation indicators, which include one or more of the following: entropy value, maximum prediction probability, vector distance from the sample to its cluster center, neighborhood label consistency rate, number of label function conflicts, sample source credibility, and historical review results.

7. The construction method according to claim 5, characterized in that, The construction method also includes: Update the version of at least one of the following objects: encoder, annotation model, first set of thresholds, and second set of thresholds; For samples that are carefully retained, the processing steps affected by the version update are re-executed, depending on the object of the version update. These processes include one or more of unified semantic coding, pre-labeling, cluster noise detection, and splitting, thereby splitting the sample again based on prediction confidence and semantic consistency.

8. The construction method according to claim 7, characterized in that, The construction method also includes: The encoder is updated by masking features at certain time steps in the feature sequence and performing self-supervised pre-training on the encoder based on the reconstruction task of the masked content. Updating the annotation model includes: training the annotation model using trusted samples, which include expert-confirmed samples and automatically confirmed samples; The first and second sets of thresholds are updated, including dynamically adjusting the thresholds based on model version, sample category distribution, expert workload, and business risk.

9. A system for constructing a network dataset, characterized in that, The construction system includes: The data access and standardization module is used to receive raw network data, process the raw network data, and obtain standardized feature sequences. The semantic representation module is used to perform unified semantic encoding on the standardized feature sequence using the encoder to obtain the semantic vector corresponding to each sample. The pre-labeling module is used to perform pre-labeling on samples based on semantic vectors using a labeling model, wherein the labeling model integrates at least two types of supervision sources, uniformly calibrates the multi-source supervision information and outputs the class probability, thereby generating temporary labels and their prediction confidence for the samples; The noise detection module is used to construct a local semantic structure through clustering based on semantic vectors, and to perform clustering noise detection on pre-labeled samples to determine semantic consistency, wherein the semantic consistency characterizes the degree of consistency between the temporary label and the local semantic structure. The confidence-based triage module is used to triage pre-labeled samples into three categories: automatically confirmed, cautiously retained, and expert-reviewed, based on predicted confidence and semantic consistency. The triage module is used to handle various types of samples differently based on the triage results.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the construction method according to any one of claims 1 to 8.

Images