Multimodal network security data automatic labeling method and system

Abstract

The application provides a kind of multi-modal network security data automatic labeling method and system based on active learning, comprising: four major core steps of multi-modal data preprocessing, model and labeling mode matching based on initialization decision algorithm, interactive labeling of "human in loop" labeling engine, iterative optimization of active learning repair engine. By designing differentiated processing logic for structured logs, text, images and other multi-modal data, combining confidence feedback, Top-K sampling, expert consensus verification and clustering assisted verification mechanism, the automation and intelligent iteration of the labeling process are realized. The application significantly improves the efficiency and quality of network security data labeling, reduces labor costs, and the model capacity can continue to grow with the data processing process, suitable for network security situation awareness and risk prevention and control scenarios of key information infrastructure such as electric power.

Description

Technical Field

[0001] This invention relates to the field of network security data processing technology, and more specifically, to a method and system for automated annotation of multimodal network security data based on active learning. Background Technology

[0002] Power grid companies have established an intelligent defense system that is regionalized, compliant with laws and regulations, trustworthy and interconnected, and coordinated in defense. They have also built a network and information security risk monitoring and early warning platform and promoted the construction of a full-scenario situational awareness platform. This provides support for situational monitoring, correlation analysis, incident handling, investigation and tracing, and intelligence gathering for the routine safe operation of various units, and can meet the needs of routine safe operation.

[0003] However, the number of high-intensity attacks, such as those related to cybersecurity drills in the power industry, is increasing. External threats are evolving rapidly, and new attack methods are constantly emerging. The resulting cybersecurity data varies significantly in format, type, and category. In recent years, artificial intelligence technology has matured, and enterprises are planning to integrate powerful machine learning models suitable for power companies to support intelligent data analysis. This work often relies on high-quality, meticulously labeled raw data, requiring significant investment in data engineering. However, the sheer volume of data generated daily makes traditional manual labeling extremely slow, resource-intensive, and error-prone—a bottleneck. This is not merely an efficiency issue; it is a key obstacle to application deployment. Automatically labeled samples do not simply require more effort in manual labeling; rather, they intelligently achieve labeling through automation.

[0004] Patent document CN115333768A discloses a method, system, and medium for analyzing massive amounts of network attack data. The method includes: automatically extracting features from massive amounts of network attack data; using algorithms to filter the most valuable attack data based on attribute features and submitting it for manual analysis; labeling the data based on the manual analysis results; calculating the similarity between the labeled data and the remaining network attack data; automatically labeling similar network attack data; and further filtering dissimilar data until all data is labeled.

[0005] However, patent document CN115333768A is designed for single-type data and lacks the ability to adapt to multimodal data. Furthermore, the isolated tree algorithm proposed in this patent document aims to select the most valuable samples. However, this method cannot provide the model with the necessary feedback. The model's capabilities are based on pre-defined rules, including tree depth and splitting node positions. It lacks the "human-in-the-loop" process of providing feedback to the model, and the model's capabilities cannot grow with the data analysis process. It does not consider reinforcement learning-related technologies.

[0006] Patent document CN120110806A discloses an automated and scalable multi-method annotation method for network traffic data, belonging to the fields of network communication, network security and artificial intelligence technology. This application supports multiple annotation methods such as protocol annotation, unsupervised annotation, feature rule annotation, graph rule annotation and agent annotation, and the annotation methods are scalable. Experts can customize annotation rules and agent annotation behaviors as needed to achieve customized multi-label annotation of network traffic data.

[0007] However, although patent document CN120110806A supports multiple annotation methods, it is not designed for the attack assessment needs of network security scenarios, and the annotation results are difficult to be directly applied to security incident analysis.

[0008] Therefore, there is an urgent need in the market for a cybersecurity data annotation solution that is adaptable to multimodal data, combines the efficiency of automation with the accuracy of human intervention, and can be iteratively optimized. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and apparatus for automated annotation of multimodal network security data based on active learning.

[0010] A method for automated annotation of multimodal network security data according to the present invention includes: Step S1: Perform targeted preprocessing on the input multimodal network security data to obtain standardized data; the multimodal network security data includes structured security log data, network security text data, and network security image data; Step S2: Based on the type of preprocessed data and the type of annotation task, select the appropriate processing model and the degree of annotation automation through a two-layer judgment logic; the two-layer judgment logic includes data type judgment and annotation task type judgment. Step S3: Generate initial annotation results and confidence probabilities using the selected model, filter samples according to confidence level for manual review and correction, and feed the correction results back to the model; Step S4: Select high-confidence error samples from the model as difficult samples, complete secondary labeling using the quality assurance mechanism, and retrain the model using the labeled data for labeling.

[0011] Preferably, the data preprocessing for structured logs includes: extracting fixed-format fields using regular expressions; performing one-hot vectorization or numerical replacement on the fixed-format information; retaining the corresponding key information using principal component analysis; and directly retaining the threat type and threat tag fields for subsequent text semantic understanding. The fixed-format fields include timestamps and IP addresses, and the key information includes IP addresses, geographical locations, and carriers.

[0012] Preferably, the network security text data includes email text and situation reports; Data preprocessing for email text data includes: constructing black email corpora and white email corpora, and using regular expressions to extract the body content of the target emails based on the basic format of the email protocol; The preprocessing of situation report data includes: using historical source analysis reports as a situation corpus to extract observable data on security incidents.

[0013] Preferably, the network security image data includes grayscale images and RGB images, and the preprocessing steps include: converting malicious program executable files into grayscale images or RGB images, and constructing a corresponding image sample library; for email data analyzed in real time, extracting executable file attachments and converting them into image format.

[0014] Preferably, the data type determination is as follows: For text data, determine whether it has a clear pattern, whether named entity recognition is required, and whether semantic analysis is required, and select regular expression extraction, pre-trained / fine-tuned small model, or advanced semi-automated model accordingly; For image data, determine whether instance segmentation and fine-grained classification are required, and select manual intervention or professional model accordingly. Task type identification: Determine whether the task is a classification task, detection / localization task, image segmentation task, or relation extraction task, which correspond to highly automated, semi-automated, computationally intensive automated, and semi-automated annotation modes, respectively.

[0015] Preferably, step S3 includes: Initial annotation generation: The appropriate model is called according to the decision result. The large language model processes text data, and the small model processes image data. The annotation results are output in a preset JSON format, which includes data content, annotation labels and confidence probability of preset intervals. Sample screening and manual review: Samples with confidence scores below the threshold or high model discrepancies are prioritized for screening based on confidence probability. The proportion of manual review shall not exceed 5% of the total samples. Experts correct the selected samples and clearly mark the error type and reason. Model feedback update: Manually corrected samples are fed back into the model as the basis for model optimization.

[0016] Preferably, step S4 includes: Hard sample selection: The Top-K method is used to select convincingly incorrect samples that have a high confidence probability in the model but are manually judged as wrong, and these samples are used as hard samples for subsequent annotation. Quality Assurance Verification: A combined QA verification method is adopted, which includes a combination of consensus voting and random auditing; clustering algorithms are used to learn the features of some manually labeled samples to automatically verify the labeling accuracy of the remaining samples; Model iterative training: The validated labeled data is added to the training set, and the automated labeling model is retrained to improve the model's ability to label edge samples and complex data.

[0017] Preferably, the Top-K method follows a data selection strategy that tends to show data annotators automatically labeled samples that are more likely to deceive humans, i.e., samples of automatically labeled incorrect answers that are convincing. Among them, convincing refers to automatically labeled samples based on confidence score probabilities, while incorrect answers refer to automatically labeled samples whose answers are ultimately determined to be incorrect by the data annotators.

[0018] Preferably, in the consensus voting stage: for key data, an odd number of expert annotators label the same Top-K items, and the result of majority voting or expert arbitration is used as the final judgment result; The random audit phase involves randomly selecting a certain percentage of labeled data for review by senior expert annotators or domain experts.

[0019] An automated annotation system for multimodal network security data provided by the present invention includes: Module M1: Performs targeted preprocessing on the input multimodal network security data to obtain standardized data; the multimodal network security data includes structured security log data, network security text data, and network security image data; Module M2: Based on the type of preprocessed data and the type of annotation task, it selects the appropriate processing model and the degree of annotation automation through a two-layer judgment logic; the two-layer judgment logic includes data type judgment and annotation task type judgment. Module M3: Generates initial annotation results and confidence probabilities using the selected model, filters samples according to confidence level for manual review and correction, and feeds the correction results back to the model; Module M4: Selects high-confidence error samples from the model as hard samples, performs secondary labeling using a quality assurance mechanism, and retrains the model using the labeled data for further labeling.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention designs different data processing logics and branch routes for multimodal network security data, and implements the most suitable data processing method under resource constraints based on the principle of "choosing the lowest option over the highest", which is of application significance in the data initialization stage of network security.

[0021] 2. This invention utilizes prompt word engineering and combines it with cybersecurity knowledge to require the automated labeler (large language model) to output the results of automatic sample labeling according to a predefined fixed format, rather than teaching it new skills. This method proposes to return the confidence probability in a predefined fixed format and apply the Top-K method to produce the samples that the automated labeler (large language model) considers to be the most correct. These samples are then judged and corrected by humans, enabling rapid iteration of the model's labeling capabilities.

[0022] 3. This invention automatically generates the most worthy examples for manual annotation. Combined with expert voting and consensus mechanisms, it ensures the accuracy of human feedback in active learning. It also uses clustering algorithms as a means of checking the automated annotator to determine whether the automated annotator is accurate. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the network security data classification process in this invention; Figure 3 This is a schematic diagram of the network security data processing flow in this invention; Figure 4 This is a schematic diagram of the "human in the loop" network security data annotation process in this invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0025] This invention employs the "Human-in-the-loop" (HITL) approach, describing a semi-automated and HITL-based automated labeler, where most of the work is automated, and human expertise assists in the active learning framework for correction.

[0026] Example 1 According to the present invention, an automated annotation method for multimodal network security data is provided, such as... Figure 1 As shown, it includes: Step S1: Multimodal network security data preprocessing. For example... Figure 2As shown, targeted preprocessing operations are performed on the input structured security log data, network security text data, and network security image data to obtain standardized data.

[0027] Structured security logs typically represent detailed alert information contained in a security incident. They originate from the collision between the security device's parsing of raw traffic and its own rule base. According to the definition of the Network Security Threat Information Format Specification (GB / T 36643-2018), they describe the attack characteristics of a single IP address and malicious domain name, determine whether the IP address is a malicious attack source, and combine external threat intelligence to form comprehensive labeled data.

[0028] Data preprocessing for structured logs includes: using regular expressions to extract fixed-format fields such as timestamps and IP addresses from the log data; vectorizing the fixed-format information in a single log entry, using a combination of one-hot vectors and numerical replacement; using principal component analysis from statistical analysis to extract the IP address, geographic location, and ISP as the main key information for a single log entry; and retaining fields such as threat type and threat tag without vectorizing them, as a finely tuned small model will be used later to understand the textual meaning of the threat type field in the logs.

[0029] The cybersecurity text data includes email text and situation reports. In phishing email attacks, the email body is usually closely related to business operations and contains a large amount of text content. Determining whether this text content uses spoofed language can effectively identify the subject of phishing emails. Situation reports typically describe in detail a particular attack activity. According to the Cybersecurity Threat Information Format Specification (GB / T 36643-2018), each attack activity includes the threat actor initiating the activity, the associated security event, and the relevant attack methods used. The security event also includes corresponding attack indicators and related observable data.

[0030] Data preprocessing for email text data includes: using historical email analysis cases to screen out a large-scale corpus of spam and phishing emails for use in bag-of-words models or word segmentation models (e.g., FastText models); using historical email analysis cases to screen out a corpus of legitimate emails based on the characteristics of business emails for use in bag-of-words models or word segmentation models; and using regular expressions to extract the body content of target analysis emails based on the basic format of email protocols (e.g., MIME, PGP, SMTP, POP3, IMAP, etc.). Data preprocessing for situation reports includes: using historically compiled source tracing analysis reports as a situation corpus for subsequent large-scale language model inference and summarization; and using regular expressions to convert situation reports into observable data for a single security event, such as an IP address, DNS domain name, or file, for use in the screening list.

[0031] The network security image data includes grayscale images and RGB images. For grayscale images, according to the Network Security Threat Information Format Specification (GB / T 36643-2018), the threat information model is divided into three domains (object domain, method domain, and event domain) and eight components. Observable objects include computer processes and executable files. This invention converts individual executable files (binary) into independent grayscale images, specifically malicious program executable files, making malware of the same family visually similar in image texture. This method can be used to distinguish malware belonging to the same family. Currently, these executable files need to be converted into grayscale images one by one and distinguished based on virus family, harm type, and scope of impact before they can be used as training datasets for different deep learning models. Open-source datasets, such as Virus-MNIST, Malimg, and SFC datasets, are widely used. For RGB images, the grayscale images of malware only contain three color types: black, white, and gray. If malware is converted into RGB images, deep learning models can effectively distinguish texture similarities and differences using the three primary colors of RGB images.

[0032] The grayscale image preprocessing steps include: converting malware and malicious executable files found in historical analysis cases into grayscale images for use as a malware image library; and extracting executable file attachments from real-time email data and converting them into image format for use as input data for machine learning models. The RGB image preprocessing steps are similar to the grayscale image preprocessing steps.

[0033] Step S2: Multimodal Data Initialization Decision: Based on the preprocessed data type and annotation task type, a suitable processing model and annotation automation level are selected through a two-layer judgment logic. The two-layer judgment logic includes data type judgment and annotation task type judgment, such as... Figure 3 As shown.

[0034] Data type determination: For text data, determine whether it has a clear pattern, whether named entity recognition is needed, and whether semantic analysis is needed, and select regular expression extraction, pre-trained / fine-tuned small model, or advanced semi-automated model accordingly; for image data, determine whether instance segmentation and fine-grained classification are needed, and select manual intervention or professional model accordingly.

[0035] Specifically, for text data: The first layer determines whether the text field has a clear pattern (e.g., the attack source IPv4 address, IPv6 address, timestamp, etc. in network attack security logs, the sender and recipient addresses in email security logs). If so, simple rule-based extraction is initiated, directly using regular expressions for extraction. If not, the second layer of judgment logic is initiated. The second layer determines whether it has Named Entity Recognition (NER) or specific common entity recognition categories (e.g., attacker tactical descriptions with clear intent in situation reports, threat types in multi-source intelligence, geographical names of attack sources, etc.). If so, a pre-trained or fine-tuned small model is selected for processing. If not, the third layer of judgment logic is initiated. The third layer determines whether sentiment analysis, opinion mining, abstract summarization, semantic recognition, etc., are required (e.g., the subject of malicious emails, phishing email content text, etc.). If so, an advanced semi-automated model or significant human supervision is used. If not, it returns that this type of text data cannot be processed.

[0036] For image data, the first layer is to perform instance segmentation and fine-grained classification of numerous small objects (e.g., grayscale images and RGB images of malicious code). If such cases exist, significant human intervention or models are required to initiate the process.

[0037] Task type determination: This determines whether the task is a classification task, detection / localization task, image segmentation task, or relation extraction task, corresponding to highly automated, semi-automated, computationally intensive automated, or semi-automated annotation modes, respectively. Multimodal data initialization decision algorithm selects the level of automation based on the annotation task type, using four conditional statements for judgment: Condition 1 is to determine whether the labeling task belongs to the classification task and whether the label belongs to one of the predefined categories (for example, email security logs, to determine whether it is spam or not, network attack logs, to determine whether it is a real attacker or an automated scanner). These are usually highly automated.

[0038] Condition 2 is to determine whether the annotation task belongs to the detection or localization type: determine whether the target is the existence and location of the object to be identified? (e.g., the bounding box of an image). Usually, this type of task is very suitable for object detection models, in a semi-automated way.

[0039] Condition 3 is to determine whether the annotation task belongs to the image segmentation category: whether it involves pixel-level objects or fixed regions. Because these annotation tasks are typically computationally intensive, requiring supercomputing hardware and complex models for automation, they are less common in cybersecurity. However, the multimodal data initialization decision algorithm described in this invention can be applied to image classification, so this invention also elaborates on one branch here.

[0040] Condition 4 is to determine whether the labeling task belongs to the relation extraction category: whether the task requires identifying relationships between entities (e.g., "The APT group White Elephant sent spear-phishing emails," "The APT group Manglinghua controlled user terminals"). These entity relationships are related to attribution analysis techniques in cybersecurity. Applying reinforcement learning to construct process-based reward functions would be more complex, so converting them into data labeling tasks is usually semi-automated, requiring the addition of a "human-in-the-loop" mindset correction.

[0041] This invention proposes an initialization decision algorithm for multimodal network security data. Based on the modality type and complexity of the initial input network security data, it automatically proposes corresponding model strategies and selects an appropriate level of automation according to the suitable annotation task type, as shown in Table 1.

[0042] This invention first determines the specific data type (text, image) and then selects a suitable model for automated annotation based on the characteristics of the data type. The data initialization decision algorithm here differs from algorithms proposed in existing patents. First, it considers the characteristics of various types of data in the cybersecurity field to "prescribe the right medicine." Second, the principle of automated model annotation is "choose the least complex," meaning that the algorithm prioritizes lighter processing methods or models based on data complexity, rather than selecting more complex models or processing algorithms. This is not merely a matter of processing efficiency, but a fundamental principle most advantageous for deployment and application.

[0043] Step S3: "Human in the Loop" Interactive Annotation. Initial annotation results and confidence probabilities are generated using the selected model. Samples are then filtered by confidence level for manual review and correction, and the corrections are fed back to the model. This invention proposes an automated annotator that optimizes annotation results by focusing expert attention on the most valuable marginal cases or cases with low confidence scores, thus transforming human experience into machine experience. Figure 4 As shown, step S3 includes: Initial Label Generation: Based on the decision results, the appropriate model is invoked. The large language model processes text data, and the small model processes image data. The labeling results are output in a preset JSON format, including data content, label labels, and confidence probabilities (0.000-0.999) within a preset range. Specifically, the decision algorithm is initialized based on multimodal network security data, processing security logs according to rules: processing malicious file images based on machine learning strategies, processing phishing email text annotation based on the large language model (LLM), and processing situation reports based on the large language model (LLM) and named entity recognition (NER) models, etc., and the initial model set is run to generate the first round of labels. This involves formatted output methods using specific response templates, representative domain-specific context insertion methods, etc., while providing auxiliary large language models to initially determine the basic knowledge structure involved, providing contextual content and representative output examples to improve the automated labeling capabilities of the large language model. The confidence probability feedback based on automated annotators is for prioritizing manual review. The automated annotator (a tool combination of a large language model, a named entity recognition (NER) classifier, and a machine learning model) should insert a confidence probability into its return request, enabling self-evaluation of its annotation process. For example, in phishing email text recognition, the large language model outputs a probability of correct analysis; the named entity recognition (NER) classifier outputs a probability for each assessed category; and the machine learning model for malicious files (grayscale images) outputs the probability of whether a file is malicious. Experts can prioritize the output annotation results based on the probabilities, allowing for review of high-probability samples and providing feedback on samples the automated annotator judged inaccurately. This enables the model to learn from inaccurate samples and improve the accuracy of subsequent confidence probability scoring.

[0044] Sample selection and manual review: Samples are sorted by confidence probability, with priority given to samples with confidence levels below the threshold or high model disagreement. The proportion of samples reviewed manually does not exceed 5% of the total sample size. Experts correct the selected samples, clearly indicating the type and reason for any errors. This invention prioritizes samples with the highest human disagreement on the model, ensuring that human effort is focused on the samples most valuable for model improvement.

[0045] Model feedback update: Manually corrected samples are fed back into the model as the basis for model optimization.

[0046] Step S4: Active Learning and Iterative Optimization. High-confidence error samples are selected as hard samples using the Top-K method. Secondary labeling is then performed using a quality assurance mechanism. The model is retrained using the labeled data to iteratively improve labeling capabilities. Figure 4 The aforementioned step S4 includes: Hard Sample Selection: The Top-K method is used to select "convincingly incorrect samples" with high confidence probabilities from the model but manually judged as errors. These are then used as hard samples for subsequent annotation. Active learning-based hard sample annotation improves the model's ability to learn from boundary samples. Iterating the model through active learning is a common and efficient tuning method in machine learning. This embodiment selects samples with high PRM scores and incorrect inferences by the data annotators from the currently learned Top-K samples with the highest confidence probabilities, using them as samples for the next round of annotation. At this stage, the invention provides more annotation information for reference, including "written solution steps," PRM scores, etc., while reducing annotation complexity. The method requires annotators to stop annotating after the first incorrect step, thus improving the efficiency of sample annotation and the quality of hard samples.

[0047] Top-K Method: In the second phase of data collection, this invention proposes a Top-K automatic annotation sample display method to improve the efficiency of human feedback after data annotators provide model judgments. Unlike traditional automatic annotation strategies that strive for even display of large language models, this invention proposes a Top-K method that filters out the most convincing error schemes to reduce the display of obviously erroneous automatically labeled samples and increase the value of the obtained human feedback.

[0048] The main processing logic of the Top-K method follows a data selection strategy that tends to show automatically labeled samples that are more likely to deceive humans, i.e., samples with convincingly incorrect answers. In this invention, "convincing" refers to automatically labeled samples based on confidence score probabilities, while "incorrect answers" refers to automatically labeled samples whose answers are ultimately determined to be incorrect by the data annotator. This invention uses "convincingly incorrect answer automatically labeled samples" for emphasis. In this embodiment, correctness is determined only by the final answer checked by the data annotator (cybersecurity expert). This invention aims to obtain more information about where the automated annotator (including large language models) goes wrong by labeling convincingly incorrect answers, thereby optimizing the inference process of the automated annotator.

[0049] In addition to this selection strategy, this invention also retrains the proposed large language model specifically for cybersecurity at multiple points during the data collection process using the latest data iterations. In each iteration, the large language model generates every annotation for a massive amount of annotation problems and only shows the data annotators the automatically annotated samples with the top K high confidence probability scores.

[0050] Quality Assurance Verification: A combined quality assurance (QA) verification method is adopted, mainly including a combination of two modes: "consensus voting" and "sampling audit". An odd number of experts vote on key samples, and the majority result is taken; samples are randomly selected for senior expert audit; clustering algorithms are used to learn the features of 5% of manually labeled samples to automatically verify the labeling accuracy of the remaining 95% of samples. Specifically, the clustering algorithm can be used to learn the features of the remaining 5% of samples without manual verification, and then label the remaining 95% of samples to determine whether the results of the automated labeler are correct.

[0051] I. Consensus Voting Phase: For key data, an odd number of expert annotators label the same Top-K items, and the result of majority voting or expert arbitration is used as the final judgment result; II. Randomized Auditing Phase: A certain percentage of the labeled data is randomly selected for review by senior expert annotators or domain experts.

[0052] Based on the combination of methods one and two, the Top-K scheme was evaluated a second time, ensuring that samples with model errors were manually annotated a second time. This combination of "consensus voting" and "spot auditing" was used to balance the results of data evaluation by multiple cybersecurity experts and maintain the accuracy of the second manual annotation results.

[0053] Model Iterative Training: The validated labeled data is added to the training set to retrain the automated labeling model, improving its ability to label marginal samples and complex data, and reducing subsequent manual intervention. Specifically, new manually labeled results after passing the aforementioned quality assurance (QA) combined validation method are collected and corrected, feeding back into the training data of the original machine learning model and the phishing email LLM model upon which the automated labeling relies. This continuous learning cycle makes the aforementioned machine learning model and phishing email LLM model increasingly accurate. Over time, the model errors detected by the Top-K method and the quality assurance (QA) combined validation method will decrease, further reducing manual workload. The above method constitutes the entire process of the active learning method proposed in this invention.

[0054] In summary, this invention utilizes a data feature dimensionality reduction method for network security alarm logs. This method addresses the high dimensionality and repetition of data features, achieving dimensionality reduction and determining which data to reduce and which to retain. It proposes an algorithmic strategy for selecting multimodal network security data. Regarding "human-in-the-loop" annotation, a high-quality "human-in-the-loop" annotation engine framework is constructed, incorporating the "human-in-the-loop" annotation engine and an active learning repair engine. For multimodal network security data, an automated annotation method based on a Large Language Model (LLM) is designed, including: a formatted output method using specific response templates and a representative domain-specific context insertion method. This integrates human feedback into the annotation process, improving the efficiency of the annotation engine. In terms of active learning repair, a data sampling method is used. After the initial annotation engine (LLM) annotates the first batch of data, the Top-K method is applied to select marginal cases or low-confidence prediction cases judged by the model, effectively identifying the first batch of annotated data requiring expert active learning. Furthermore, it elucidates an improved training method for active learning and proposes a quality control (QA) mechanism based on confidence probability scores. Using the confidence scores of the second-stage "annotation engine" as a foundation, a combination of public voting and random auditing (randomly sampling and manually checking 5% of the annotations, and using clustering algorithms to accelerate manual verification) is employed to correct the judgment results of the automated annotation method, ensuring that the automated annotation results are as consistent as possible with human expectations and improving the accuracy of automated annotation.

[0055] Example 2 This invention's data screening engine includes the classification of multimodal network security data and a multimodal data initialization decision algorithm. The following two specific data engineering examples illustrate the specific methods for processing network security data.

[0056] For example, applying "Man in a Ring" (HITL) to grayscale image classification of malware. Initial scan: Train a deep learning model (e.g., Minst-Virus) on a small number of manually labeled grayscale images of malicious code (e.g., Trojans, worms).

[0057] Batch preprocessing: Run Minst-Virus on 2,000 new grayscale images. The model outputs suggested malware types and calculates a confidence score for each image.

[0058] Priority: Minst-Virus models with confidence scores below a certain threshold (e.g., 0.7) are pushed to the top of the human review queue.

[0059] Manual correction: Cybersecurity experts use annotation tools to refine the analysis logic, correct boundary errors, or add missing analytical thinking within the model.

[0060] Feedback and Retraining: Corrected new grayscale images are added to the training set. The Minst-Virus model is periodically retrained on an expanded dataset to improve its ability to judge future grayscale images and reduce the number of low-confidence predictions.

[0061] For example, in the automated analysis of network security logs, it is necessary to detect whether each log entry was initiated by a real attacker. This is a common and well-defined task with clear analysis steps, making it a typical case of automation. Since network security logs contain two types of text, such as attack source and timestamp text with fixed formats, which can be directly processed using regular expressions, but for threat types and tags in attack logs, large language models and Named Entity Recognition (NER) technology are required. Therefore, a combination of multiple strategies is needed.

[0062] Example 3 The sample engineering process based on an automated labeler proposed in this invention is as follows: Initialize and insert specific domain context: This section combines internal enterprise emergency plans and collects 16 key security incident emergency response scenarios (handling successful attack events, handling attacks exceeding thresholds, handling zero-day vulnerability risk warnings, handling DDoS attack behavior, handling sensitive confidential data outgoing, one-click firewall shutdown, handling honeypot and encrypted network attack events, handling abnormal access to malicious domains such as mining and ransomware, handling phishing and virus emails, handling weak email account passwords, handling suspected controlled email accounts, handling abnormal email account login behavior, handling mobile terminal security monitoring, handling business system failures, handling API interface abnormal alarms, and handling terminal alarms from desktop management systems and antivirus software). The generated knowledge serves as a basic knowledge base in a specific template format.

[0063] Output method using a specific response template: It is required to use a large language model to assist in labeling phishing texts, and provide output in a fixed format when analyzing entity relationships in situation reports through Named Entity Classification (NER). For example, when a large language model assists in labeling phishing texts, a few-shot prompt can be constructed and a fixed return format can be specified, such as the prompt as follows: "You are a Chinese or English email phishing information classifier. Given an email, classify it as 'phishing email' or 'normal email'. Meanwhile, provide a confidence probability score (from 0.000 to 0.999). Any email containing bad words, slang, inducing fraudulent intent, or potential cyberattacks in Chinese or English shall be regarded as a phishing email. The confidence probability score indicates your degree of confidence that this text is a phishing email. Highly phishing information shall have a higher confidence probability value. Use Chinese and English language knowledge to identify phishing email texts. Please follow the exact response pattern. Provide me with JSON key-value pairs, do not give me any other text. The response shall be (ensure that strings are enclosed in double quotes and the JSON format is valid) <return>{ "message": the body of the given phishing email; "phishing": 1 if it is phishing, 0 if it is not phishing; "confidence probability": the confidence score for the judgment on whether it is phishing (0.000 to 0.999); }< / return>" The present invention only automatically labels data that meets the cybersecurity data classification criteria in phase one, and complies with the above return format.

[0064] The small-volume data labeling process with human participation includes a cold start labeling stage. Currently, there is only the result of an automatic labeler outputting in JSON format. Using the initial version of labeled samples, labeling personnel need to sort the initial labeling results according to the returned confidence probability, select the Top-K labeling results as the final high-quality output. If no correct execution result is finally provided, the labeling personnel shall perform manual editing to correct the correct "yes" labeling results in these Top-K samples. Due to the limited availability of expert manpower, this method stipulates that the proportion of manually labeled samples shall not exceed 5% of the total sample size.

[0065] Embodiment 4 The present invention further provides a multimodal cybersecurity data automatic labeling system. The multimodal cybersecurity data automatic labeling system can be implemented by executing the process steps of the multimodal cybersecurity data automatic labeling method, that is, those skilled in the art can regard the multimodal cybersecurity data automatic labeling method as a preferred embodiment of the multimodal cybersecurity data automatic labeling system.

[0066] An automated annotation system for multimodal network security data provided by the present invention includes: Module M1: Performs targeted preprocessing on the input multimodal network security data to obtain standardized data. The multimodal network security data includes structured security log data, network security text data, and network security image data. Preprocessing of the structured log data includes: extracting fixed-format fields using regular expressions; vectorizing the fixed-format information using one-hot vectorization or numerical replacement; retaining corresponding key information using principal component analysis; and directly retaining threat type and threat tag fields for subsequent text semantic understanding. The fixed-format fields include timestamps and IP addresses, and the key information includes IP address, geographical location, and carrier.

[0067] Module M2: Based on the type of preprocessed data and the type of annotation task, a two-layer judgment logic selects the appropriate processing model and the degree of annotation automation. The two-layer judgment logic includes data type judgment and annotation task type judgment. The network security text data includes email text and situation reports. Data preprocessing for email text data includes: constructing black email corpora and white email corpora, and using regular expressions to extract the body content of the target analysis emails according to the basic format of email-specific protocols. Data preprocessing for situation reports includes: using historical source tracing analysis reports as a situation corpus to extract observable data of security events. The network security image data includes grayscale images and RGB images. The preprocessing module includes: converting malicious program executable files into grayscale or RGB images and constructing corresponding image sample libraries. For email data analyzed in real time, executable file attachments are extracted and converted into image format. Data type judgment: For text data, it sequentially judges whether it has a clear pattern, whether named entity recognition is needed, and whether semantic analysis is needed, and selects regular expression extraction, pre-trained / fine-tuned small model, and advanced semi-automated model accordingly. For image data, determine whether instance segmentation and fine-grained classification are needed, and select manual intervention or a professional model accordingly. Labeling task type determination: determine whether the task is a classification task, detection / localization task, image segmentation task, or relation extraction task, corresponding to highly automated, semi-automated, computationally intensive automated, and semi-automated labeling modes, respectively.

[0068] Module M3: Generates initial annotation results and confidence probabilities using the selected model, filters samples based on confidence levels for manual review and correction, and feeds the correction results back to the model. Module M3 includes: Initial Annotation Generation: Based on the decision result, the appropriate model is called; a large language model processes text data, and a small model processes image data. Annotation results are output in a preset JSON format, including data content, annotation labels, and confidence probabilities for preset intervals. Sample Screening and Manual Review: Samples are sorted by confidence probability, prioritizing those with confidence levels below a threshold or high model divergence. The manual review rate does not exceed 5% of the total samples. Experts correct the selected samples, clarifying the type and cause of annotation errors. Model Feedback Update: The manually corrected samples are fed back to the model as basic data for model optimization.

[0069] Module M4: This module selects high-confidence error samples as hard samples, performs secondary labeling using a quality assurance mechanism, and retrains the model using the labeled data. Module M4 includes: Hard Sample Selection: Using a Top-K method, it selects convincing error samples with high model confidence probability but manually judged as incorrect, using these as hard samples for subsequent labeling. Quality Assurance Verification: Employing a combined quality assurance (QA) verification method, including consensus voting and random auditing. Clustering algorithms are used to learn some manually labeled sample features, automatically verifying the labeling accuracy of the remaining samples. Model Iterative Training: The verified labeled data is added to the training set, and the automated labeling model is retrained to improve its ability to label marginal samples and complex data. The Top-K method follows a data selection strategy that tends to show data annotators samples that are more likely to deceive humans, i.e., automatically labeled samples with convincing incorrect answers. Here, "convincing" refers to automatically labeled samples based on confidence score probabilities, and "incorrect answers" refers to automatically labeled samples whose answers are ultimately determined to be incorrect by the data annotator. The consensus voting phase involves having an odd number of expert annotators label the same Top-K items for critical data, and using the majority vote or expert arbitration as the final judgment. The random audit phase involves randomly selecting a certain proportion of labeled data for review by senior expert annotators or domain experts.

[0070] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, enabling the system and its various devices, modules, and units to function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0071] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for automated annotation of multimodal network security data, characterized in that, include: Step S1: Perform targeted preprocessing on the input multimodal network security data to obtain standardized data; The multimodal network security data includes structured security log data, network security text data, and network security image data; Step S2: Based on the type of preprocessed data and the type of annotation task, select the appropriate processing model and the degree of annotation automation through a two-layer judgment logic; the two-layer judgment logic includes data type judgment and annotation task type judgment. Step S3: Generate initial annotation results and confidence probabilities using the selected model, filter samples according to confidence level for manual review and correction, and feed the correction results back to the model; Step S4: Select high-confidence error samples from the model as difficult samples, complete secondary labeling using the quality assurance mechanism, and retrain the model using the labeled data for labeling.

2. The automated annotation method for multimodal network security data according to claim 1, characterized in that, Data preprocessing for structured logs includes: extracting fixed-format fields using regular expressions; vectorizing fixed-format information using one-hot vectors or numerical replacement; retaining corresponding key information through principal component analysis; and directly retaining threat type and threat tag fields for subsequent text semantic understanding. The fixed-format fields include timestamps and IP addresses, and the key information includes IP addresses, geographical locations, and carriers.

3. The automated annotation method for multimodal network security data according to claim 1, characterized in that, The cybersecurity text data includes email texts and situation reports; Data preprocessing for email text data includes: constructing black email corpora and white email corpora, and using regular expressions to extract the body content of the target emails based on the basic format of the email protocol; The preprocessing of situation report data includes: using historical source analysis reports as a situation corpus to extract observable data on security incidents.

4. The automated annotation method for multimodal network security data according to claim 1, characterized in that, The network security image data includes grayscale images and RGB images. The preprocessing steps include: converting malicious program executable files into grayscale images or RGB images and constructing a corresponding image sample library; for email data analyzed in real time, extracting executable file attachments and converting them into image format.

5. The automated annotation method for multimodal network security data according to claim 1, characterized in that, Data type determination: For text data, determine whether it has a clear pattern, whether named entity recognition is needed, and whether semantic analysis is needed, and select regular expression extraction, pre-trained / fine-tuned small model, or advanced semi-automated model accordingly; For image data, determine whether instance segmentation and fine-grained classification are needed, and select manual intervention or professional model accordingly. Task type identification: Determine whether the task is a classification task, detection / localization task, image segmentation task, or relation extraction task, which correspond to highly automated, semi-automated, computationally intensive automated, and semi-automated annotation modes, respectively.

6. The automated annotation method for multimodal network security data according to claim 1, characterized in that, Step S3 includes: Initial annotation generation: The appropriate model is called according to the decision result. The large language model processes text data, and the small model processes image data. The annotation results are output in a preset JSON format, which includes data content, annotation labels and confidence probability of preset intervals. Sample screening and manual review: Samples with confidence levels below the threshold or high model discrepancies are prioritized for screening based on confidence probability. The proportion of manual review shall not exceed 5% of the total samples. Experts correct the selected samples and clearly mark the error type and reason. Model feedback update: Manually corrected samples are fed back into the model as the basis for model optimization.

7. The automated annotation method for multimodal network security data according to claim 1, characterized in that, Step S4 includes: Hard sample selection: The Top-K method is used to select convincingly incorrect samples that have a high confidence probability in the model but are manually judged as wrong, and these samples are used as hard samples for subsequent annotation. Quality Assurance Verification: A combined QA verification method is adopted, which includes a combination of consensus voting and random auditing; clustering algorithms are used to learn the features of some manually labeled samples and automatically verify the labeling accuracy of the remaining samples; Model iterative training: The validated labeled data is added to the training set, and the automated labeling model is retrained to improve the model's ability to label edge samples and complex data.

8. The automated annotation method for multimodal network security data according to claim 7, characterized in that, The Top-K method follows a data selection strategy that tends to show data annotators automatically labeled samples that are more likely to deceive humans, i.e., samples of automatically labeled incorrect answers that are convincing. Among them, convincing refers to automatically labeled samples based on confidence score probabilities, while incorrect answers refer to automatically labeled samples whose answers are ultimately determined to be incorrect by the data annotators.

9. The automated annotation method for multimodal network security data according to claim 7, characterized in that, The consensus voting phase: For key data, an odd number of expert annotators label the same Top-K items, and the result of majority voting or expert arbitration is used as the final judgment result; The random audit phase involves randomly selecting a certain percentage of labeled data for review by senior expert annotators or domain experts.

10. A multimodal network security data automated annotation system, characterized in that, include: Module M1: Performs targeted preprocessing on the input multimodal network security data to obtain standardized data; The multimodal network security data includes structured security log data, network security text data, and network security image data; Module M2: Based on the type of preprocessed data and the type of annotation task, it selects the appropriate processing model and the degree of annotation automation through a two-layer judgment logic; the two-layer judgment logic includes data type judgment and annotation task type judgment. Module M3: Generates initial annotation results and confidence probabilities using the selected model, filters samples according to confidence level for manual review and correction, and feeds the correction results back to the model; Module M4: Selects high-confidence error samples from the model as hard samples, performs secondary labeling using a quality assurance mechanism, and retrains the model using the labeled data for further labeling.

Images