Network threat intelligence structured processing method and system based on artificial intelligence

Abstract

The invention relates to the technical field of crossing of artificial intelligence and network security, in particular to a network threat intelligence structured processing method and system based on artificial intelligence. The method comprises the following steps: acquiring unstructured original threat intelligence and performing credibility evaluation on an intelligence source; based on a large language model and knowledge graph enhanced retrieval, a step-by-step analysis plan for original threat intelligence is generated; dynamically calling a tool to execute an information extraction subtask corresponding to each step in the analysis plan to obtain a preliminary extraction result; performing multi-dimensional confidence evaluation on the preliminary extraction result; and taking the preliminary extraction result with the confidence higher than or equal to a threshold as a final result, and outputting the final result in a formatting manner and updating the final result to the knowledge graph. According to the method, the problem that unstructured multi-mode threat intelligence is difficult to convert into structured data which is readable by a machine, self-consistent in logic and capable of being put into actual combat in the prior art is solved, and the improvement of attack chain understanding depth and integrity is realized.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and cybersecurity, and in particular to a method and system for structured processing of network threat intelligence based on artificial intelligence. Background Technology

[0002] Cyber ​​threat intelligence (CTI) is the cornerstone of modern proactive defense systems. Intelligence sources are diverse, including PDF analysis reports released by security vendors, blog posts from technical communities, open-source intelligence sources (OSINT), and dark web forums. This intelligence contains crucial information such as attackers' tactics, techniques, processes (TTPs), tools used, and attack metrics (IoC). However, the vast majority (over 90%) of CTI exists in unstructured text (PDF, HTML) and image formats, heavily relying on security analysts for manual reading, understanding, extraction, and formatting. This process is not only inefficient but also highly error-prone, resulting in a significant amount of valuable intelligence being buried or delayed in its use.

[0003] To address this issue, the industry has explored various automation technologies. Early solutions primarily relied on regular expressions and keyword matching, resulting in low accuracy and poor generalization. Subsequently, deep learning technologies, represented by pre-trained language models such as BERT and RoBERTa, were widely applied, achieving Named Entity Recognition (NER) and Relation Extraction (RE) through fine-tuning on labeled corpora. Currently, a representative solution is to introduce a hybrid architecture of LLM and deep learning to extract unstructured network threat intelligence. However, this is essentially a simple superposition of existing tools / models, with the following drawbacks: (a) It still employs an "end-to-end" or "flat" processing flow, lacking a structured and interpretable task decomposition and planning mechanism; (b) It fails to effectively address the knowledge illusion and knowledge lag issues of LLM in specialized domains; (c) It lacks a systematic logical verification and cross-modal consistency check mechanism; and (d) It has not established a closed-loop continuous learning and evolution mechanism.

[0004] In summary, the current cybersecurity field faces a massive amount of unstructured, multimodal threat intelligence (such as APT analysis reports, security blogs, incident announcements, etc.), which is difficult to interpret effectively. Summary of the Invention

[0005] To overcome the problem in the existing technology of converting unstructured multimodal threat intelligence into machine-readable, logically consistent, and combat-ready structured data, this invention proposes an artificial intelligence-based method for structuring network threat intelligence, which can process unstructured raw threat intelligence into structured data.

[0006] This invention proposes a structured processing method for network threat intelligence based on artificial intelligence, comprising the following steps: S1. Obtain unstructured raw threat intelligence and assess the credibility of the intelligence source; S2. Enhanced retrieval based on large language models and knowledge graphs to generate a step-by-step analysis plan for the original threat intelligence; S3. Dynamically call the tool to execute the information extraction sub-tasks corresponding to each step in the analysis plan to obtain preliminary extraction results; S4. Perform multi-dimensional confidence assessment on the preliminary extraction results; trigger iterative calibration when the confidence level is lower than the threshold; the iterative calibration method is as follows: obtain the low confidence dimension of the low confidence result and the corresponding reason for questioning; feed the low confidence result together with the low confidence dimension and the reason for questioning back to the large language model and the corresponding tools, so that the large language model and the corresponding tools take the reason for questioning as a constraint condition, return to step S3, and obtain the preliminary extraction results again; S5. Use the preliminary extraction results with confidence levels higher than or equal to the threshold as the final results, format the final results, and update them to the knowledge graph.

[0007] Preferably, step S2 specifically involves: retrieving background knowledge related to the intelligence topic from the knowledge graph, injecting it into the prompt template of the large language model in a structured manner, and guiding the large language model to generate an analysis plan.

[0008] Preferably, the method for multi-dimensional confidence evaluation of the preliminary extraction results in step S4 is as follows: calculate the confidence of the preliminary extraction results on the specified confidence dimensions, and then sum them by weight to obtain the total confidence of the preliminary extraction results; the specified confidence dimensions include semantic consistency. Logical consistency Domain knowledge fit and cross-modal consistency Multiple types.

[0009] Preferably, in step S1, the method for obtaining unstructured raw threat intelligence is as follows: configure keywords and behavioral patterns, and combine them with a distributed crawler framework to crawl raw threat intelligence; the crawling frequency is directly proportional to the activity level of the intelligence source.

[0010] Preferably, in step S3, the tool router is set to resolve and analyze the plan, and the tool function corresponding to each step is obtained; the tool router maps the steps to the corresponding tool function call instructions and parameters and executes them to obtain the preliminary extraction results corresponding to each step.

[0011] Preferably, the tools include multiple of the following: time-series relation extraction tools, entity extraction tools, relation construction tools, attack framework mapping tools, and formatting tools.

[0012] Preferably, in S4, if the number of iterations for the initial extraction results reaches a set number and the total confidence level is still lower than the threshold, the original threat intelligence and related information are pushed to the manual review queue to obtain the processed manual correction results and correction instructions. The related information includes the latest initial extraction results, low confidence dimensions, and reasons for questioning. The original threat intelligence, manual correction results, and correction instructions are used as high-quality samples for incremental fine-tuning of the large language model. In S4, new knowledge is also extracted from high-quality samples and injected into the knowledge graph.

[0013] Preferably, after step S4 and before step S5, causal attribution is performed on the final result. The attention weight or attribution algorithm inside the large language model is used to extract the input text from the original text fragment, calculate the contribution of words / sentences in the input text to the generation of the final result, and use the original text fragment with the highest contribution as evidence to support the final result.

[0014] Preferably, after step S4 and before step S5, a hypothesis is generated for the final result. This is done by: masking or modifying some of the evidence input, rerunning part of the analysis process, observing and outputting the changes in the final result, and forming multiple "main hypothesis-alternative hypothesis" chains of evidence that are aligned with each other.

[0015] This invention proposes a structured processing system for network threat intelligence based on artificial intelligence, comprising: The intelligence acquisition module is used to proactively acquire raw threat intelligence from multiple intelligence sources and assess the credibility of these sources. The Mind Chain Planning module is used to receive raw threat intelligence and generate an analysis plan containing multiple ordered steps based on a large language model and knowledge graph. The information extraction module is used to parse the analysis plan and dynamically call the corresponding tools in the tool library to execute each step to generate preliminary extraction results; The verification and calibration module is used to evaluate the confidence of the preliminary extraction results in at least two dimensions, and trigger iterative calibration when the confidence is lower than the threshold. The original threat intelligence and related information that have reached the set number of iterations and whose total confidence is still lower than the threshold are pushed to the manual review queue to obtain the processed manual correction results and correction instructions. The related information includes the latest preliminary extraction results, low confidence dimensions and reasons for questioning. The knowledge accumulation module is used to update the knowledge graph with the original threat intelligence after manual review, as well as the manual correction results and correction instructions.

[0016] The advantages of this invention are: (1) The present invention proposes a structured processing method for network threat intelligence based on artificial intelligence. By combining large language models and knowledge graphs to deeply understand the original threat intelligence and generate multi-step analysis plans, it realizes the planning of thought chains, enabling the system to perform task decomposition and deep context association like a human analyst, effectively solving the problem of cross-paragraph / chapter dependencies in long documents. The present invention achieves a leap in the depth and completeness of attack chain understanding.

[0017] (2) This invention realizes dynamic trial spectrum through knowledge graph update, ensuring that the latest background knowledge is integrated into the intelligence analysis process in real time, and effectively suppresses the knowledge illusion of large language models through retrieval enhancement, effectively solving the knowledge lag problem and improving the recognition accuracy of new TTPs (attack processes).

[0018] (3) This invention combines semantic consistency Logical consistency Domain knowledge fit and cross-modal consistency Multi-dimensional cross-validation is performed across equal dimensions to achieve industrial-grade reliability and logical self-consistency assurance. Furthermore, this invention, combined with a self-consistent calibration micro-loop, provides a systematic and automated correction capability.

[0019] (4) This invention makes the conclusion “verifiable” through causal tracing, which greatly enhances the analyst’s trust; the generation of counterfactual hypotheses actively provides alternative perspectives, helping analysts break through the “tunnel vision”, discover potential overlooked threats, and improve the quality of decision-making.

[0020] (5) The active learning closed loop of this invention automatically transforms manual review feedback into high-quality training data and knowledge graph updates, driving continuous incremental optimization of the core model and knowledge base, and achieving "the more you use it, the smarter it becomes". The system maintenance mode changes from the traditional high-cost and long-cycle "full retraining" to efficient "incremental fine-tuning", and can quickly adapt to changes in the threat landscape, achieving self-evolution and low-cost maintenance.

[0021] (6) This invention utilizes large-scale language models (such as LLM) to automate and structure the extraction, logical verification, and dynamic construction of knowledge graphs of multi-source, multimodal, and unstructured network threat intelligence (CTI). It realizes an innovative architecture that integrates thought chain task planning, dynamic knowledge graph enhanced retrieval (KG-RAG), adaptive tool scheduling, and multi-dimensional cross-validation. Its core lies in building a closed-loop intelligent processing system of "cognition-execution-verification-evolution" to achieve a technological breakthrough in threat intelligence from passive processing to active hunting and from static extraction to dynamic evolution.

[0022] (7) The present invention proposes a network threat intelligence structured processing system based on artificial intelligence, which integrates active hunting, intelligent analysis, rigorous verification and closed-loop evolution, and realizes end-to-end high automation from original intelligence discovery to high-value structured knowledge output and application. It realizes active defense, frees analysts from the heavy work of information extraction, focuses on high-level analysis and judgment, and significantly improves the foresight of threat response.

[0023] (8) Researchers observed from the trial data that: the invention significantly improves the completeness of attack chain reconstruction from about 58% in the prior art to more than 94%, achieving a complete characterization of complex APT attacks; the invention significantly reduces the logical consistency error rate (about 15%) that is common in the prior art to below 0.5%, achieving a high level of reliability for the automated output structured data that can be directly used for automated response in the Security Operations Center (SOC); and after adopting the invention, the life cycle maintenance cost of the system is reduced by about 70%. Attached Figure Description

[0024] Figure 1 This is a flowchart of a structured processing method for network threat intelligence based on artificial intelligence proposed in this invention; Figure 2 Flowchart for intelligence gathering; Figure 3 This is the step-by-step process for step S2; Figure 4 The flowchart for the information extraction subtask in step S3 is as follows; Figure 5 A flowchart for multi-dimensional verification and iterative verification; Figure 6 Flowcharts are generated for causal attribution and hypothesis testing; Figure 7 To incorporate manual adjustments and process optimization. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0026] Reference Figure 1 The present invention proposes a structured processing method for network threat intelligence based on artificial intelligence, comprising the following steps: S1. Obtain unstructured raw threat intelligence and assess the credibility of the intelligence source; like Figure 2As shown, this step can be achieved by defining an intelligent agent to proactively acquire raw threat intelligence from intelligence sources and assess the credibility of those sources. This upgrades the system from a passive intelligence processing mode to a proactive intelligence discovery and assessment mode, thus solving the fundamental problems of unreliable intelligence sources and delayed intelligence acquisition.

[0027] Intelligence sources include, but are not limited to, specific dark web forums, technical community blogs, open-source code repositories, and social media channels of known attack groups. This method defines an agent as the system's "outpost," responsible for proactively and continuously monitoring high-value intelligence sources on the internet, aiming to discover emerging threats, vulnerabilities, or attack tools as soon as possible.

[0028] This step achieves high-concurrency crawling by configuring keywords (e.g., 0-day, new malware family, data leak) and behavioral patterns (e.g., a known threat actor releasing a new executable file), combined with a distributed crawling framework (such as Scrapy-Redis). Near real-time monitoring is achieved through a WebSocket / API long polling mechanism (for dynamic intelligence sources such as dark web forums). The crawling frequency is dynamically adjusted according to the activity level of the intelligence source (10 minutes / time for active intelligence sources, 2 hours / time for inactive intelligence sources). Once the preset rules (i.e., keywords and behavioral patterns) are triggered, the relevant intelligence is immediately crawled and sent to subsequent analysis processes.

[0029] In this step, after acquiring intelligence, the credibility of the intelligence's "source" is first quantitatively assessed, and then the intelligence content is analyzed in depth. This approach addresses the widespread problem of false intelligence, false alarms, and even "polluted intelligence" intended to mislead in cyberspace. In this step, raw threat intelligence from high-credibility sources can directly proceed to the next step; raw threat intelligence from low-credibility sources is either discarded or labeled as low-credibility before being sent to the next step.

[0030] In practice, a dynamic "intelligence source credibility knowledge base" can be pre-set. For each intelligence source, a credibility score can be calculated based on factors such as historical data (e.g., the proportion of intelligence it publishes that has been subsequently verified) and community cross-validation (e.g., whether multiple independent high-credibility intelligence sources report the same event). It is worth noting that, because intelligence sources release information dynamically, their credibility score varies. It is also dynamic and changing.

[0031] S2. Enhanced retrieval based on large language models and knowledge graphs generates analysis plans for raw threat intelligence. This transforms raw, unstructured intelligence into machine-executable, logically clear analysis blueprints, overcoming the limitations of flat processing.

[0032] like Figure 3 As shown, this step first retrieves background knowledge related to the intelligence topic (i.e., the topic of the original threat intelligence) from the dynamic knowledge graph (KG), such as attacker tactical and technical processes (TTPs), tools, vulnerabilities, organizational information, etc.; the background knowledge is then structured and injected into the prompt template of the large language model (e.g., the LLM's Prompt template), guiding the large language model LLM to generate a step-by-step, interpretable "analysis plan" (thinking chain), rather than directly outputting the extraction results, for example: "1. Identify and extract all IoCs in the report; 2. Reconstruct the timeline of the attack activities; 3. Correlate and map the attack behaviors to the latest version of the ATT&CK framework; 4. Verify the logical relationships (temporal, causal) between entities."

[0033] Background knowledge includes known attack tactics (TTPs), commonly used tools, vulnerability information, and historical event patterns. This step guides the LLM (Local Master Module) with background knowledge from the KG (Key Knowledge Base) subgraph information structure, making the generated analysis plan more domain-specific and in-depth, effectively suppressing illusions. For example, before analyzing the APT41 report, information such as APT41's known TTPs (MITRE ATT&CK ID) and commonly used tools is injected.

[0034] In this step, a specific Prompt template is designed to guide the LLM in task decomposition and planning, which further facilitates the LLM's understanding of background knowledge.

[0035] S3. Analyze the analysis plan and dynamically call tools to execute the information extraction subtasks corresponding to each step in the analysis plan to obtain preliminary extraction results. Specifically, in this step, each step is treated as an information extraction subtask. The configured tool router parses the analysis plan to obtain the tool function corresponding to each step, and maps the steps to the corresponding tool function call instructions and parameters, so that each tool executes the information extraction subtask corresponding to each step.

[0036] like Figure 4 As shown, this step precisely translates each step of the analysis plan into actual extraction actions, thereby enabling the scheduled execution of the analysis plan.

[0037] This step first parses the "analysis plan" and, based on the instructions for each step of the "analysis plan," dynamically and sequentially calls the most suitable specialized tool functions from the tool library. For example, when the planned step is "Reconstruct Timeline," the system automatically calls the "Time Series Relationship Extraction Tool" to execute the step. The tool library includes, but is not limited to, time series relationship extraction tools, entity extraction tools, relationship building tools, ATT&CK mapping tools, IoC formatting tools, etc. This "deconstruct before execution" model maps planned steps to specific tool call instructions and parameters, breaking down a complex extraction task into a series of simpler and more precise subtasks.

[0038] S4. Perform multi-dimensional confidence assessment on the preliminary extraction results, obtain the final results with confidence levels higher than or equal to the threshold, and trigger iterative calibration when the confidence level falls below the threshold. The essence of this step is "fact checking," which aims to ensure that all extraction results have extremely high accuracy and logical consistency before being put into use, so as to guarantee the quality and reliability of the output results.

[0039] like Figure 5 As shown, this step involves a rigorous quantitative evaluation of the initial results returned by each tool call. The evaluation is based on four independent dimensions: semantic consistency ( Logical consistency Domain knowledge fit ) and cross-modal consistency ( ) .

[0040] Total confidence level It is composed of four weighted dimensions:

[0041] in: (Semantic consistency): A contrastive learning model (Siamese Network / SimCSE, etc.) finely tuned on the CTI corpus is used to calculate the semantic similarity between the extracted results, i.e. the preliminary extracted results, and the supporting evidence fragments in the original text, to prevent misinterpretation. (Logical Consistency): Built-in rule-based logical reasoning engine automatically checks sequential logic, causal logic, and type constraints. (a) Temporal logic can be represented as time correlation, such as: intrusion time < lateral movement time < data theft time; (b) Causal logic can be represented as conditional association, such as: malware must exploit a specific vulnerability to achieve its effect; (c) Type constraints can be set based on experience, such as: IP addresses cannot have version numbers, CVE numbers must conform to a format, etc. (Domain Knowledge Compliance): Extracted TTPs, tools, vulnerabilities, etc., are matched with the dynamically updated MITREATT&CK vector knowledge base to assess their relevance within the current threat framework. The knowledge base is updated near real-time with ATT&CK updates; (Cross-modal consistency): When processing reports containing images / screenshots, use visual models (ViT, CLIP, etc.) to analyze the image content and verify whether the text description (such as "screenshot shows Cobalt Strike beacon") is consistent with the image information.

[0042] Weighting coefficient The settings are based on a dynamic strategy: the base values ​​are optimized using the MITRE ATT&CK case set; The base weights are determined by optimizing the MITRE ATT&CK case set, for example, they can be set as follows: =0.3, =0.4, =0.2, =0.1). The weight can be based on the credibility of the intelligence source ( Dynamic adjustment (e.g., low) Intelligence, Improve The weight of (logical consistency) (Weight of cross-modal consistency).

[0043] In step S4, when the total confidence level ( When the confidence level is below a preset threshold (e.g., 0.9), a self-consistent calibration micro-loop (i.e., iterative calibration) is automatically triggered. This is done by obtaining the low-confidence dimension and corresponding reasons for doubt in the preliminary extraction results (referred to as low-confidence results) where the confidence level is below the threshold. The low-confidence dimension is defined as the dimension less than the set threshold. (semantic consistency) (Logical consistency) (Domain knowledge fit) and (Cross-modal consistency); the reason for the questioning is the low-confidence triggering condition corresponding to the low-confidence dimension; for example... The low confidence dimension is " The reason for the questioning, "(Logical consistency) is below the corresponding threshold," is "A time sequence conflict was detected: Event A should be later than Event B." Then, the low confidence result is followed by the specific low confidence dimension and the reason for the questioning (e.g., "..."). Low: A temporal conflict was detected: Event A should be later than Event B. This is used as a new constraint and fed back to the Large Language Model (LLM) and the corresponding tools (i.e., the tools used in this analysis). The LLM and the corresponding tools must re-analyze and generate new preliminary extraction results while satisfying this constraint. This "generate-verify-correct" cycle can be iterated several times until the total confidence level is reached. Greater than or equal to the set threshold If the number of iterations reaches the maximum number of iterations, but the total confidence score is low... Still below the set threshold If so, it will be pushed to the manual review queue.

[0044] S5. Format the final results with confidence levels higher than or equal to the threshold and update them to the knowledge graph.

[0045] In this step, the final results with a confidence level higher than or equal to the threshold can be output in a specified format (such as the STIX 2.1 standard) and synchronously updated to the dynamic knowledge graph (KG). Updates include: adding entities / relationships / attributes, supplementing event instances, associating ATT&CK technologies, and updating TTP descriptions, etc.

[0046] This step updates the knowledge graph based on the final results, consolidating all new knowledge gained during the analysis process, ensuring the system's continuous and self-optimizing capabilities, and enabling the system to memorize and learn.

[0047] For the final result with a confidence level below the threshold, i.e., the preliminary extraction result where the number of iterations and calibrations in step S4 reaches the set number and the total confidence level is still below the threshold, the original threat intelligence, along with the final result (i.e., the latest preliminary extraction result), the low-confidence dimension, and the reason for the questioning, can be pushed to the manual review queue. Security analysts will review and correct this, obtaining corresponding high-quality samples, which may include the original threat intelligence, the manual correction result, and the correction explanation. In this embodiment, the corrected high-quality samples can also be automatically absorbed as training data to periodically perform incremental fine-tuning (Delta Tuning) of the LLM based on the training data, verifying the contrastive learning model / logic rule weights, etc., in the LLM model.

[0048] like Figure 7 As shown in this embodiment, high-quality samples can be further processed manually to extract new knowledge (such as new TTPs, new tools, and new organizations) from the high-quality samples and inject it into the knowledge graph to dynamically update the knowledge graph, thereby forming a closed-loop evolutionary process of "processing -> verification -> manual intervention -> learning -> improvement".

[0049] After step S4 and before step S5, this embodiment also performs causal tracing and hypothesis generation on the final result to solve the "black box" problem of the AI ​​decision-making process, so that the system can not only give a "credible" answer, but also provide an "understandable" reasoning process, thus deepening the level of human-machine collaboration.

[0050] Specifically, the causal attribution method for the final result is as follows: For each final result (e.g., mapping the attack behavior to a certain ATT&CK technology ID), the attention weights or attribution algorithms (SHAP, LIME) within the large language model are used to calculate the contribution of words / sentences in the input text to the generation of the final result. The input text may come from intelligence text or text recognized from intelligence images through OCR. The original text fragment (text or image) with the highest contribution is highlighted as text or image evidence supporting the final result.

[0051] In practice, the evidence obtained from causal tracing can be processed into an evidence chain report and visualized.

[0052] The method for generating hypotheses about the final result is as follows: after generating the final result, some key evidence inputs are programmatically masked or modified (such as ignoring the attribution information of a certain IP or assuming that the vulnerability has not been exploited), some analysis processes are rerun (such as ATT&CK mapping and attack chain reconstruction), the changes in the final result are observed and output, multiple "master hypothesis-alternative hypothesis" are aligned with their respective evidence chains, and the results are output in a structured manner.

[0053] It is evident that when faced with complex scenarios where intelligence content is ambiguous and evidence is insufficient, hypothesis generation can go beyond a single conclusion and generate multiple "alternative analytical hypotheses" with reasonable basis to assist human experts in making decisions and avoid cognitive biases.

[0054] This invention also proposes an automated network threat intelligence processing system based on artificial intelligence, which includes: an intelligence acquisition module, a thought chain planning module, an information extraction module, a verification and calibration module, a knowledge accumulation module, and an interpretation enhancement module.

[0055] The intelligence gathering module is used to proactively acquire raw threat intelligence from multiple sources and assess the credibility of these sources. Specifically, the module includes an autonomous hunting agent and a source evaluation engine. The autonomous hunting agent is configured to monitor and crawl internet intelligence sources at dynamically adjusted frequencies using a distributed crawling framework and long polling mechanism, based on preset keywords and behavioral patterns. The source evaluation engine is configured to calculate and output dynamic credibility scores based on the historical accuracy of the intelligence sources and cross-validation information from the community.

[0056] The Mind Chain Planning module receives raw threat intelligence and generates an analysis plan containing multiple ordered steps based on a large language model and knowledge graph.

[0057] The information extraction module is used to parse the analysis plan and dynamically call the corresponding tools in the tool library to execute each step to generate preliminary extraction results.

[0058] The verification and calibration module is used to evaluate the confidence level of the preliminary extraction results in at least two dimensions and trigger iterative calibration when the confidence level falls below a threshold. Specifically, in this embodiment, the verification and calibration module includes a four-dimensional verification engine configured to calculate the total confidence level of the preliminary extraction results from four specified dimensions: semantic consistency, logical consistency, domain knowledge compliance, and cross-modal consistency. The weights of each dimension of the total confidence level are dynamically adjusted based on the confidence score output by the source evaluation engine.

[0059] The knowledge accumulation module updates the knowledge graph with results whose confidence level exceeds a threshold, thus achieving a closed loop of active learning and evolution. The knowledge accumulation module is configured as follows: Results with a confidence level below the threshold and that fail the calibration cycle are pushed to the manual review queue. Receive manual corrections and correction explanations from security analysts regarding the results; Data pairs containing original threat intelligence, manual correction results, and correction instructions are used as training samples to incrementally fine-tune the large language model or validation model in the system. New knowledge confirmed in the manual correction results is injected into the knowledge graph.

[0060] The explanation enhancement module includes a causal attribution unit and a counterfactual hypothesis generator. The causal attribution unit is configured to use attribution algorithms to locate and highlight textual or image evidence in the original intelligence that supports key conclusions; the counterfactual hypothesis generator is configured to rerun part of the analysis process by procedurally modifying key evidence in the original threat intelligence to generate at least one alternative analytical hypothesis and its chain of evidence.

[0061] The following describes the above-mentioned artificial intelligence-based structured processing method for network threat intelligence (hereinafter referred to as the present invention method) in conjunction with specific embodiments.

[0062] To verify the effectiveness, reliability, and advantages of this invention compared to existing technologies, this experiment constructed a test dataset covering multi-source, multi-modal threat intelligence. The data samples were derived from users' historical data, and a comparative experiment was conducted with existing mainstream solutions. The experimental environment was a GPU server (NVIDIA A40×8), and the software environment consisted of Python 3.10 + PyTorch 2.0 + HuggingFace Transformers.

[0063] This embodiment uses two comparative models to compare the effects with the method of the present invention.

[0064] Comparison Model 1 employs the BERT-BiLSTM-CRF model architecture based on the BERT-FT algorithm, fine-tuned on the CTI-Bench training set. The BERT-FT algorithm represents the mainstream deep learning-based threat intelligence extraction paradigm from 2019 to 2023. Its core idea is to utilize a BERT model pre-trained on a general corpus, and then perform supervised fine-tuning on domain-specific, manually labeled threat intelligence data to adapt it to entity recognition and relation extraction tasks in the cybersecurity field.

[0065] Model 2, in comparison, adopts a hybrid architecture of "LLM coarse extraction + small model refinement" proposed in CN118013046A. This represents an emerging approach that appeared after 2023, attempting to improve CTI processing by leveraging the general capabilities of Large Language Models (LLMs). Its core idea is to use the strong generalization ability and zero-shot / few-shot learning characteristics of LLMs (such as GPT-3.5 / 4) for "coarse extraction," and then use a lightweight, accurate, specialized small model (such as BERT) for "refinement," forming a two-stage pipeline to reduce reliance on manual rules.

[0066] Thus, after training Contrast Model 1 on the CTI-Bench training set, it is tested on the test dataset along with Contrast Model 2 and the method of this invention. It is worth noting that neither the method of this invention nor Contrast Model 2 requires training; LLM directly uses a pre-trained model and is then executed directly on the test data.

[0067] The following evaluation metrics are used in this embodiment: Entity Extraction F1-Score (F1-Entity): A metric that comprehensively evaluates a system's ability to identify and classify key entities (such as malware, attack groups, tools, vulnerabilities, IoCs, etc.) in threat intelligence, and can fully reflect the system's extraction performance. Relation Extraction Accuracy (RE-Acc): Evaluates the system's ability to correctly infer and classify semantic relationships between correctly identified entity pairs; Relation Extraction Accuracy = (Number of correctly predicted relation triples) / (Total number of predicted relation triples) × 100%; Hallucination Rate (HR): Directly quantifies the severity of "creation out of thin air" in the content generated by the large language model; Hallucination Rate = (Number of "fictitious" entities or relations generated by the system that have no basis in the original text) / (Total number of all entities and relations generated by the system) × 100%; Chain Completeness (CC): Assessing the system's ability to understand the global attack narrative; Chain Completeness (CC) = (Number of attack steps correctly reconstructed by the system) / (Total number of attack steps annotated by experts) × 100%; New TTP Accuracy (NTA): Evaluates the system's accuracy in identifying threat techniques not present in the training set; New TTP Accuracy (NTA) = (Number of new TTPs correctly identified by the system) / (Total number of new TTPs in the test set) × 100%; Logical Consistency Error (LCE): Evaluates the proportion of logical inconsistencies (such as time reversal or causal errors) in the output results; Logical Consistency Error Rate (LCE) = (Number of output entries containing logical errors) / (Total number of system output entries) × 100%; Human Review Rate (HRR): Used to measure the automation level and usability of a system; Human Review Rate (HRR) = (Number of output cases with confidence levels below the threshold requiring manual correction) / (Total number of cases processed by the system) × 100%.

[0068] The evaluation results of this embodiment are shown in Table 1 below.

[0069] Table 1: Comparison of Model Performance

[0070] Experimental results show that the method of this invention achieves significant improvements in attack chain reconstruction integrity (CC) and novel TTP recognition accuracy (NTA), reaching 89.7% and 79.3%, respectively. Simultaneously, thanks to the four-dimensional verification engine, the logical consistency error rate (LCE) is reduced to an extremely low level of 4%. In fine-grained extraction tasks, the entity extraction F1 score (F1-Entity) and relation extraction accuracy (RE-Acc) reach 92.5% and 91.0%, respectively, and the illusion rate (HR) is effectively suppressed to 4.2%, demonstrating the high reliability of the system output.

[0071] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0072] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0073] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A structured processing method for network threat intelligence based on artificial intelligence, characterized in that, Includes the following steps: S1. Obtain unstructured raw threat intelligence and assess the credibility of the intelligence source; S2. Enhanced retrieval based on large language models and knowledge graphs to generate a step-by-step analysis plan for the original threat intelligence; S3. Dynamically call the tool to execute the information extraction sub-tasks corresponding to each step in the analysis plan to obtain preliminary extraction results; S4. Conduct a multi-dimensional confidence assessment of the preliminary extraction results; When the confidence level is lower than the threshold, iterative calibration is triggered. The iterative calibration method is as follows: obtain the low confidence dimension of the low confidence result and the corresponding reason for questioning; feed the low confidence result together with the low confidence dimension and the reason for questioning back to the large language model and the corresponding tool, so that the large language model and the corresponding tool take the reason for questioning as the constraint condition, return to step S3, and re-obtain the preliminary extraction result. S5. Use the preliminary extraction results with confidence levels higher than or equal to the threshold as the final results, format the final results, and update them to the knowledge graph.

2. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 1, characterized in that, Step S2 specifically involves retrieving background knowledge related to the intelligence topic from the knowledge graph, injecting it into the prompt template of the large language model in a structured manner, and guiding the large language model to generate an analysis plan.

3. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 1, characterized in that, The method for multi-dimensional confidence evaluation of the preliminary extraction results in step S4 is as follows: calculate the confidence of the preliminary extraction results on the specified confidence dimensions, and then sum them with weights to obtain the total confidence of the preliminary extraction results; the specified confidence dimensions include semantic consistency. Logical consistency Domain knowledge fit and cross-modal consistency Multiple types.

4. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 1, characterized in that, In step S1, the method for obtaining unstructured raw threat intelligence is as follows: configure keywords and behavioral patterns, and combine them with a distributed crawler framework to crawl raw threat intelligence; the crawling frequency is directly proportional to the activity of the intelligence source.

5. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 1, characterized in that, In step S3, the tool router is set to resolve and analyze the plan, and the tool function corresponding to each step is obtained. The tool router maps the steps to the corresponding tool function call instructions and parameters and executes them to obtain the preliminary extraction results corresponding to each step.

6. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 5, characterized in that, The tools include: Multiple tools including time-series relation extraction tools, entity extraction tools, relation building tools, attack framework mapping tools, and formatting tools.

7. The structured processing method for network threat intelligence based on artificial intelligence as described in claim 1, characterized in that, In S4, if the number of iterations for the initial extraction results reaches the set number and the total confidence level is still lower than the threshold, the original threat intelligence and related information are pushed to the manual review queue to obtain the processed manual correction results and correction instructions. The related information includes the latest initial extraction results, low confidence dimensions, and reasons for questioning. The original threat intelligence, manual correction results, and correction instructions are used as high-quality samples for incremental fine-tuning of the large language model. In S4, new knowledge is also extracted from high-quality samples and injected into the knowledge graph.

8. The method for structured processing of network threat intelligence based on artificial intelligence as described in any one of claims 1-7, characterized in that, After step S4 and before step S5, causal attribution is performed on the final result. The attention weight or attribution algorithm inside the large language model is used to extract the input text from the original text fragment, calculate the contribution of words / sentences in the input text to the generation of the final result, and use the original text fragment with the highest contribution as evidence to support the final result.

9. The method for structured processing of network threat intelligence based on artificial intelligence as described in any one of claims 1-7, characterized in that, After step S4 and before step S5, a hypothesis is generated for the final result by: masking or modifying some of the evidence input, rerunning part of the analysis process, observing and outputting the changes in the final result, and forming multiple "main hypothesis-alternative hypothesis" to align their respective evidence chains.

10. A structured processing system for network threat intelligence based on artificial intelligence, characterized in that, include; The intelligence acquisition module is used to proactively acquire raw threat intelligence from multiple intelligence sources and assess the credibility of these sources. The Mind Chain Planning module is used to receive raw threat intelligence and generate an analysis plan containing multiple ordered steps based on a large language model and knowledge graph. The information extraction module is used to parse the analysis plan and dynamically call the corresponding tools in the tool library to execute each step to generate preliminary extraction results; the verification and calibration module is used to evaluate the confidence of the preliminary extraction results in at least two dimensions, and trigger iterative calibration when the confidence is lower than the threshold. The original threat intelligence and related information that have reached the set number of iterations and whose total confidence is still lower than the threshold are pushed to the manual review queue to obtain the processed manual correction results and correction instructions. The related information includes the latest preliminary extraction results, low confidence dimensions and reasons for questioning. The knowledge accumulation module is used to update the knowledge graph with the original threat intelligence after manual review, as well as the manual correction results and correction instructions.

Images