Threat intelligence big data governance system and method based on knowledge graph

Abstract

The invention provides a threat intelligence big data governance system and method based on a knowledge graph, and relates to the field of network security, the system comprises a data acquisition module, an entity extraction module, a knowledge graph construction module, a graph governance module, a graph evolution module and an output module; according to the application, through multi-module cooperation, multi-source heterogeneous threat intelligence is standardized, then entities are extracted and ambiguity is resolved, and a structured knowledge graph is constructed; redundant contradictions are eliminated through governance and optimization, credibility is calibrated, timeliness and accuracy of information are guaranteed in combination with time decay and a feedback mechanism, and finally multi-form output supports downstream application. The problems of single processing, no context and difficulty in distinguishing redundant contradictions in the prior art are effectively solved, the credibility, integrity and availability of threat intelligence are remarkably improved, the false alarm rate is reduced, and the data processing efficiency and the downstream security defense effectiveness are improved.

Description

Technical Field

[0001] This application relates to the field of cybersecurity, and more specifically, to a knowledge graph-based threat intelligence big data governance system and method. Background Technology

[0002] As cyberattacks evolve towards multi-stage and cross-platform approaches, relying solely on traditional security devices such as firewalls, intrusion detection systems, and antivirus software is no longer sufficient to effectively combat complex attack behaviors. To address this, the industry has proposed the concept of threat intelligence, which involves collecting information from multiple sources, including open-source intelligence, commercial intelligence sources, internal logs, and computer emergency response team reports, to analyze malicious IP addresses, domain names, and attacker organizational backgrounds, thereby enabling proactive attack prevention.

[0003] However, in practical applications, threat intelligence suffers from diverse sources and inconsistent quality: intelligence from different channels may overlap, or even contain contradictory information, and some may even contain false intelligence deployed by attackers. Existing governance technologies are mostly rule-based field standardization and simple deduplication, or statistical source reputation scoring and frequency filtering, which can only process data at the level of a single intelligence item and have not established structured relationships between intelligence items.

[0004] This makes it difficult for existing technologies to detect the differences between isolated intelligence and global information, and to identify cross-source redundancy and semantic contradictions. It also makes it easy to retain worthless data or transmit incorrect intelligence, resulting in misreporting and underreporting in security operations center decision-making, which seriously affects the effectiveness of downstream defense systems. Summary of the Invention

[0005] The purpose of this application is to provide a threat intelligence big data governance system and method based on knowledge graphs, which solves the above-mentioned problems existing in the prior art, addresses the pain points of intelligence governance, and improves intelligence credibility and processing efficiency.

[0006] Firstly, a knowledge graph-based threat intelligence big data governance system is provided, which may include: The data acquisition module is used to collect raw threat intelligence from multiple sources and heterogeneous sources, and to preprocess and standardize the raw threat intelligence to obtain threat intelligence data. The entity extraction module is used to extract entities and relationships between entities from the threat intelligence data, standardize synonymous entities, and resolve different names for the same entity. The knowledge graph construction module is used to store entities and relationships in a graph database, define the node and edge types of the graph, store attribute information in nodes and edges, and remove redundant nodes in the graph through entity alignment. The knowledge graph governance module is used to optimize the quality and calibrate the credibility of the constructed threat intelligence knowledge graph, resulting in an optimized knowledge graph. The graph evolution module is used to dynamically update the optimized knowledge graph. The update includes at least a time decay mechanism that reduces the credibility of outdated intelligence over time, and a feedback update mechanism that adjusts the graph based on external verification results to obtain a credible intelligence subgraph. The output module is used to output the processed trusted intelligence subgraph, providing interface calls, result display, and early warning.

[0007] In one possible implementation, the graph governance module includes an island detection unit for identifying isolated nodes, a redundancy merging unit for merging duplicate entities, a contradiction identification unit for discovering conflicting information, and a weight calculation unit for calculating the credibility of entities and relationships.

[0008] In one possible implementation, the data acquisition module is specifically used to extract potential indicators from the raw intelligence using regular expressions and natural language processing techniques. These potential indicators include domain names, IP addresses, file hashes, and vulnerability numbers. Standardization includes converting the preprocessed intelligence into a unified format.

[0009] In one possible implementation, the entity extraction module extracts entities through a deep learning named entity recognition model, the entities including malicious IPs, malicious domains, malicious samples, attacker organizations, and attack tools; and extracts attack path relationships between entities through dependency parsing and graph neural networks.

[0010] In one possible implementation, the entity extraction module's homonym resolution includes merging different names of the same attacker organization into the same entity.

[0011] In one possible implementation, the attribute information stored by the knowledge graph construction module includes timestamps, intelligence source weights, and initial credibility; the entity alignment is achieved through an attribute similarity algorithm, and the calculation dimensions of the attribute similarity include entity name, associated IP / domain name, historical attack records, and source overlap.

[0012] In one possible implementation, the island detection unit of the graph governance module identifies isolated nodes through connected subgraph analysis and marks isolated nodes as low-value or erroneous intelligence candidates; the redundancy merging unit aggregates duplicate entity nodes through a similarity matching algorithm.

[0013] Secondly, a knowledge graph-based approach to threat intelligence big data governance is provided, which may include: The data acquisition module collects raw threat intelligence from multiple heterogeneous sources, and performs preprocessing and standardization on the raw threat intelligence to obtain threat intelligence data. The entity extraction module extracts entities and relationships between entities from the threat intelligence data, standardizes synonymous entities, and resolves different names for the same entity. The knowledge graph construction module stores entities and relationships in a graph database, defines the node and edge types of the graph, stores attribute information in nodes and edges, and removes redundant nodes in the graph through entity alignment. The knowledge graph governance module performs quality optimization and credibility calibration on the constructed threat intelligence knowledge graph to obtain an optimized knowledge graph; The graph evolution module dynamically updates the optimized knowledge graph, including at least a time decay mechanism that reduces the credibility of outdated intelligence over time, and a feedback update mechanism that adjusts the graph based on external verification results to obtain a credible intelligence subgraph. The output module outputs the processed trusted intelligence subgraph, providing interface calls, result display, and early warning.

[0014] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the method described in the second aspect above.

[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of the method described in the second aspect above.

[0016] This application provides a threat intelligence big data governance system and method based on knowledge graphs, relating to the field of cybersecurity. The system includes a data acquisition module, an entity extraction module, a knowledge graph construction module, a graph governance module, a graph evolution module, and an output module. Through multi-module collaboration, this application first standardizes multi-source heterogeneous threat intelligence, then extracts entities and resolves ambiguities to construct a structured knowledge graph. After governance and optimization to eliminate redundancy and contradictions and calibrate credibility, time decay and feedback mechanisms ensure the timeliness and accuracy of the intelligence. Finally, multiple forms of output support downstream applications. This effectively solves the problems of single-item processing, lack of context, and difficulty in identifying redundant contradictions in existing technologies, significantly improving the credibility, completeness, and usability of threat intelligence, reducing false positive rates, and increasing data processing efficiency and the effectiveness of downstream security defenses. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A system architecture diagram of a threat intelligence big data governance system based on knowledge graphs is provided for embodiments of this application; Figure 2 A flowchart illustrating a knowledge graph-based threat intelligence big data governance method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] As cyberattacks evolve towards multi-stage and cross-platform approaches, relying solely on traditional security devices such as firewalls, intrusion detection systems, and antivirus software is no longer sufficient to effectively combat complex attack behaviors. To address this, the industry has proposed the concept of threat intelligence, which involves collecting information from multiple sources, including open-source intelligence, commercial intelligence sources, internal logs, and computer emergency response team reports, to analyze malicious IP addresses, domain names, and attacker organizational backgrounds, thereby enabling proactive attack prevention.

[0021] However, in practical applications, threat intelligence suffers from diverse sources and inconsistent quality: intelligence from different channels may overlap, or even contain contradictory information, and some may even contain false intelligence deployed by attackers. Existing governance technologies are mostly rule-based field standardization and simple deduplication, or statistical source reputation scoring and frequency filtering, which can only process data at the level of a single intelligence item and have not established structured relationships between intelligence items.

[0022] This makes it difficult for existing technologies to detect the differences between isolated intelligence and global information, and to identify cross-source redundancy and semantic contradictions. It also makes it easy to retain worthless data or transmit incorrect intelligence, resulting in misreporting and underreporting in security operations center decision-making, which seriously affects the effectiveness of downstream defense systems.

[0023] Therefore, this application provides a threat intelligence big data governance system and method based on knowledge graphs, which solves the above-mentioned problems existing in the prior art, addresses the pain points of intelligence governance, and improves intelligence credibility and processing efficiency.

[0024] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0025] Figure 1 This document presents a system architecture diagram for a knowledge graph-based threat intelligence big data governance system, as provided in an embodiment of this application. Figure 1 As shown, the system may include: a data acquisition module, an entity extraction module, a knowledge graph construction module, a knowledge graph governance module, a knowledge graph evolution module, and an output module.

[0026] A. Data acquisition module, used to collect raw threat intelligence from multiple sources and heterogeneous sources, and to preprocess and standardize the raw threat intelligence to obtain threat intelligence data.

[0027] Furthermore, multi-source heterogeneous raw threat intelligence can include open-source intelligence channels, such as public cybersecurity communities, technical blogs, and open-source threat intelligence databases; closed or semi-closed channels, such as dark web forums and covert discussions related to cyberattacks on social media; professional intelligence sharing platforms, such as threat intelligence sharing platforms (MISP) and data sources officially released by security vendors; internal enterprise data, such as network logs and endpoint behavior logs generated by the enterprise's daily operations; reports from authoritative organizations, such as security incident reports and vulnerability warning documents issued by the Computer Emergency Response Team (CERT); and commercial intelligence sources, namely proprietary threat intelligence data obtained from compliant commercial intelligence service providers.

[0028] For real-time data, Kafka can be used as a real-time data bus. Its high throughput and low latency characteristics enable the rapid collection of time-sensitive intelligence such as dynamic information from dark web forums, real-time discussions on social media, and emergency alerts from security vendors, ensuring that critical intelligence enters the system without delay. For batch data, an architecture combining HDFS (distributed file system) and Spark (distributed computing framework) can be used to store and process massive amounts of non-real-time intelligence such as enterprise historical logs, CERT archived reports, and batch data from open-source intelligence databases. Distributed computing capabilities can improve the processing efficiency of batch data and avoid massive data backlog.

[0029] Specifically, it is used to extract potential indicators from raw intelligence using regular expressions and natural language processing techniques. These potential indicators include domain names, IP addresses, file hashes, and vulnerability numbers. Standardization includes converting the pre-processed intelligence into a unified format.

[0030] Preprocessing can use filtering rules to remove invalid characters, duplicate text, advertising information and other interfering content from the original intelligence, while retaining the core information related to network attacks and malicious behavior. The entity extraction process can combine regular expressions and natural language processing (NLP) technology to accurately extract potential threat indicators from the cleaned intelligence, including domain names, IP addresses, file hash values ​​(such as MD5, SHA256), vulnerability numbers (such as CVE numbers), etc. These indicators are the core basis for subsequent identification of malicious entities and construction of threat relationships.

[0031] The format can be standardized using JSON or Avro format as the standard output format. Both formats have good readability and scalability, and can be compatible with the field storage requirements of different types of threat intelligence (such as intelligence source, collection time, key indicators, descriptive information, etc.). Field standardization can define unified field naming rules and data types (such as unifying the intelligence source field as a string type and the collection time field as a timestamp format), avoiding parsing errors in downstream modules due to non-standard fields and ensuring smooth data flow throughout the governance system.

[0032] B. Entity Extraction Module: This module extracts entities and relationships between entities from threat intelligence data, standardizes synonymous entities, and resolves different names for the same entity.

[0033] Specifically, the entity extraction module extracts entities through a deep learning named entity recognition model, including malicious IPs, malicious domains, malicious samples, attacker organizations, and attack tools; and extracts attack path relationships between entities through dependency parsing and graph neural networks.

[0034] First, the types of entities extracted from threat intelligence data include, but are not limited to: malicious network elements, attackers and tools, and victim and vulnerability information; Specifically, malicious network elements include: malicious IP addresses, malicious domain names, malicious file hash values ​​(such as MD5, SHA256), and malicious URLs; The specific targets and tools of the attack include: attacker organizations (such as APT groups), attack tools (such as Cobalt Strike), and attack tactics (such as phishing and ransomware). The victim and vulnerability information specifically includes: the affected industry / company, the vulnerability number (such as CVE number), and the vulnerability type (such as SQL injection or remote code execution).

[0035] Furthermore, the BERT+CRF deep learning model is used to complete named entity recognition: Semantic encoding of threat intelligence texts is performed based on BERT pre-trained models to capture the contextual semantic features of technical terms (such as APT28, CVE-2024-XXXX) in the text, thus solving the problem of ambiguous texts that are difficult to handle by traditional rule matching. Combining Conditional Random Fields (CRF) with sequence labeling of encoded semantic features enables precise location of entity boundaries and improves the accuracy of entity recognition, especially for distinguishing between entities with similar names (such as similar code names of different APT organizations) and different representations of the same entity (such as malicious IP and attack source IP).

[0036] Next, the logical relationships (relationships between entities) between the aforementioned entities are identified and extracted, forming a triple structure of entity, relation, and entity. Relationship types include, but are not limited to: Attribution: For example, a malicious IP address may belong to an APT group; Usage relationship: such as APT groups using attack tools; Relationships: such as malicious IP communication and malicious file hashes; Impact on related industries: such as the impact of vulnerabilities on affected industries.

[0037] Furthermore, to extract the relationships between entities, we can first use dependency parsing to analyze the grammatical structure of the intelligence text, identify the grammatical dependencies between entities in the sentence (such as subject-verb and verb-object relationships), and preliminarily screen out entity pairs that may be related. Then, graph neural networks (GNNs) are used to learn features from the initially selected entity pairs and contextual information to further determine the semantic association type between entities (such as use, attack, association), ensuring the accuracy of relation extraction and avoiding erroneous associations caused by grammatical ambiguity (such as mistakenly identifying CobaltStrike defense as CobaltStrike use).

[0038] Standardizing synonymous entities can also be understood as unifying and standardizing the different forms of the same entity due to differences in format and expression habits, so as to avoid redundant nodes in subsequent graph construction.

[0039] For example, the IP address 08.08.08.08 can be unified into 8.8.8.8 by removing the prefix zero; the domain names www.xxx.com and xxx.com (without the www prefix) can be merged into the same domain name entity according to DNS resolution rules; Unify the case of letters in the hash values ​​of malicious files (e.g., unify a1b2c3d4 and A1B2C3D4 to lowercase) and remove meaningless prefixes (e.g., remove the Hash: identifier before the hash value in some reports). For example, unifying Terminal Detection and Response and Terminal Protection Response into Terminal Detection and Response (i.e., the standard Chinese expression for EDR) ensures consistency in entity names.

[0040] The entity extraction module's homonym resolution includes merging different names for the same attacker organization into a single entity. Specifically, it involves identifying and merging different names for the same entity, thus eliminating homonyms and ensuring that the same entity corresponds to only one unique node in the knowledge graph.

[0041] For example, APT28 and Fancy Bear (different names for the same APT group in the industry), and APT1 and Unit 61398 can be grouped into the same attacker organization entity; For example, the EternalBlue vulnerability and CVE-2017-0144 can be grouped into the same vulnerability entity; Furthermore, by constructing an entity homonymous knowledge base (which includes known homonymous entity pairs), and combining semantic similarity matching algorithms (such as cosine similarity and Jaccard similarity), the entity names appearing in the intelligence are compared with the knowledge base, automatically identifying homonymous entities and merging them. At the same time, manual review and correction are supported to ensure the accuracy of the resolution results.

[0042] The entity extraction module ultimately outputs a dataset of triples of entities and relations in a uniform format. Each triple contains core information about the head entity, relation type, and tail entity, along with entity attributes (such as entity type, first extraction time, and source intelligence ID) and relation attributes (such as relation confidence and extraction basis text fragment). This dataset can be directly read by the knowledge graph construction module for subsequent graph database storage and graph structure modeling.

[0043] C. Knowledge graph construction module, used to store entities and relations to the graph database, define the node and edge types of the graph, store attribute information in nodes and edges, and remove redundant nodes of the graph through entity alignment.

[0044] Specifically, the attribute information stored in the knowledge graph construction module includes timestamps, intelligence source weights, and initial credibility; entity alignment is achieved through an attribute similarity algorithm, and the dimensions for calculating attribute similarity include entity name, associated IP / domain name, historical attack records, and source overlap.

[0045] First, to meet the needs of threat intelligence involving multi-entity relationships and highly complex queries, this module selects a professional graph database that supports efficient graph traversal and relation calculation, specifically Neo4j or JanusGraph. The selection criteria and adaptation design are as follows: Neo4j features a native graph storage architecture with fast single-node query response speed, making it suitable for the rapid construction and real-time querying of small to medium-sized threat intelligence graphs; JanusGraph supports distributed storage and multi-data source integration, which can meet the storage and computing needs of massive threat intelligence (such as tens of millions of entities / relationships) and is suitable for large-scale enterprise deployment scenarios.

[0046] In some embodiments, TigerGraph or a self-developed graph storage based on HBase / Spark may also be used.

[0047] This module has a built-in database adaptation interface, which can flexibly switch graph databases according to the system deployment scale (such as Neo4j for small and medium-sized security teams and JanusGraph for large enterprises). It also uniformly encapsulates the data writing / query API, ensuring that subsequent modules (such as graph governance and output modules) do not need to modify the code when calling graph data, thus ensuring system compatibility.

[0048] Subsequently, combining the semantic logic of threat intelligence with industry knowledge, this module standardizes the definition of node types and edge types in the graph to ensure that the graph structure meets the needs of security analysis scenarios.

[0049] Node types include: attacker organizations, attack tools, attack tactics, victim industries, vulnerabilities, IPs / domains, and malicious files.

[0050] Attacker groups refer to organizations or groups that carry out cyberattacks (such as APT groups or hacker groups). Attack tools refer to the software, scripts, or malicious code used by attackers (such as Cobalt Strike, ransomware). Attack tactics refer to the attacker's attack methods or strategies (such as phishing attacks, remote code execution, lateral movement). Victim industries / enterprises refer to industries (such as finance and energy) or specific enterprises that have been attacked or are at risk of being attacked. A vulnerability refers to an exploitable flaw in a system or software (such as a vulnerability corresponding to a CVE number or a zero-day vulnerability). IP / domain refers to network addresses associated with attacks (such as malicious IPs, malicious domains, C2 server addresses); Malicious files refer to files that exhibit malicious behavior (uniquely identified by file hash values ​​such as MD5 and SHA256).

[0051] In some embodiments, to support subsequent graph governance (such as weight calculation and contradiction identification) and evolution (such as time decay), this module stores key attribute information in the nodes and edges of the graph. The specific attributes and storage rules are as follows: Timestamp: Records the first extraction time of an entity / relationship (i.e., the time when the entity / relationship is extracted from threat intelligence), with a uniform format of YYYY-MM-DD HH:MM:SS, used by the subsequent time decay mechanism to determine the freshness of intelligence; Intelligence Source Weight: Records the basic weight of the source of intelligence to which the entity / relationship belongs (e.g., the weight of an authoritative threat intelligence platform is 0.7, and the weight of an official log source is 0.9). The weight value ranges from 0 to 1 and is used as the basis for subsequent credibility calculations. Initial credibility: The initial credibility of an entity / relationship is calculated based on the weight of the intelligence source (e.g., for an entity from a single source, the initial credibility = the weight of that source; for an entity from multiple sources, the initial credibility = the average weight of the multiple sources), which is used as the initial scoring benchmark for subsequent graph governance.

[0052] Each node and edge is associated with a unique attribute table, and the attribute information corresponds one-to-one with the ID of the node / edge. Real-time query and modification of attributes are supported (if the credibility is adjusted by the subsequent feedback update mechanism, the initial credibility attribute value can be updated directly).

[0053] Finally, to avoid redundant nodes in the graph caused by multiple sources of the same entity, this module eliminates redundancy through an entity alignment algorithm based on attribute similarity. The specific process and calculation dimensions are as follows: Obtain the set of entities to be aligned (i.e., the new entities output by the entity extraction module, which are the same type of entities that already exist in the graph). Calculate the multi-dimensional similarity between the entity to be aligned and the existing entities, and sum them to obtain the comprehensive similarity. Set a similarity threshold (e.g., a comprehensive similarity of ≥0.8 indicates the same entity). If the similarity exceeds the threshold, merge them into the same node and retain multi-source attribute information (e.g., the intelligence source weight of the merged node includes the weight values ​​of all sources). If the similarity does not exceed the threshold, determine it as a new entity and add a new node to the graph.

[0054] In some embodiments, the Jaccard similarity algorithm is used to calculate the character overlap and semantic relevance of entity names (such as the attacker organization codename APT28 and Fancy Bear, and the vulnerability name CVE-2024-XXXX and a certain remote code execution vulnerability), with a weighting of 30%. If an entity is an attacker organization, malicious file, etc., calculate the intersection ratio of its associated IP / domain sets (e.g., entity A is associated with IP set {IP1, IP2}, entity B is associated with IP set {IP1, IP3}, intersection ratio = 1 / 2 = 0.5), weighted at 25%; The overlap of historical attack targets (such as victim industries and enterprises) and attack tactics associated with entities is calculated (e.g., entity A attacks the financial industry using phishing tactics, and entity B attacks the financial industry using remote code execution; overlap = 1 / 2 = 0.5), with a weighting of 25%. Calculate the intersection ratio of the intelligence sources to which the entity belongs (e.g., entity A comes from source 1 and source 2, entity B comes from source 2 and source 3, intersection ratio = 1 / 2 = 0.5), weighted at 20%.

[0055] In some embodiments, redundancy merging may use hash matching, edit distance, or vectorized embedding similarity.

[0056] In summary, the knowledge graph construction module ultimately outputs a structured threat intelligence knowledge graph, which specifically includes: Each node contains core information such as node ID, node type, and entity name, as well as attribute information such as timestamp, intelligence source weight, and initial credibility. Each edge contains core information such as edge ID, head node ID, tail node ID, and edge type, as well as attribute information such as timestamp, intelligence source weight, and initial credibility. Record the entities to be aligned, existing entities, similarity in each dimension, overall similarity, and alignment results (merged / added) during the entity alignment process for subsequent traceability and manual review.

[0057] D. The knowledge graph governance module is used to optimize the quality and calibrate the credibility of the constructed threat intelligence knowledge graph, resulting in an optimized knowledge graph.

[0058] Specifically, the graph governance module includes an island detection unit to identify isolated nodes, a redundancy merging unit to merge duplicate entities, a contradiction identification unit to discover conflicting information, and a weight calculation unit to calculate the credibility of entities and relationships.

[0059] Furthermore, the island detection unit of the graph governance module identifies isolated nodes through connected subgraph analysis and marks isolated nodes as low-value or erroneous intelligence candidates; the redundancy merging unit aggregates duplicate entity nodes through a similarity matching algorithm.

[0060] D1. The island detection unit uses connected subgraph analysis technology to identify isolated nodes. The specific processing flow is as follows: Step 1: Using the depth-first search (DFS) or breadth-first search (BFS) algorithm of the graph database, traverse all nodes and edges in the initial knowledge graph, and divide the nodes that are related to each other (directly or indirectly) into a connected subgraph, forming multiple independent sets of connected subgraphs. Step 2: Configure the isolated node determination threshold. If a connected subgraph contains only 1 node (i.e., the node has no incoming or outgoing edges and has not established any connection with other nodes), then the node is determined to be an isolated node. Step 3: Add low-value / erroneous intelligence candidate tags to the identified isolated nodes and record the basic information of the nodes (such as node type, extraction source, and first entry time) to facilitate subsequent manual review or automatic screening and removal (for example, isolated nodes that have been marked for more than a preset period and have not been verified can be automatically stored from the graph to the set to be cleaned).

[0061] In one example, if there is a malicious IP node (IP99) in the initial graph that has no connection with any other entity such as the attacker organization, malicious files, or victims, after connected subgraph analysis, the connected subgraph containing this node only contains itself. Therefore, it is marked as a low-value / erroneous intelligence candidate. Subsequently, it can be manually verified to confirm whether the IP is invalid information that was mistakenly entered, so as to avoid it interfering with the normal threat link analysis.

[0062] D2. The redundant merging unit uses an algorithm based on multi-dimensional similarity matching to merge duplicate entities. The specific processing flow is as follows: Step 1: Entity pairing and filtering: Classify the entities in the initial graph according to node type (e.g., group IP / domain type nodes into one category and attacker organization type nodes into another category), and only perform repeated matching within the same type of node to avoid misjudgment of cross-type entities; Step 2: Multi-dimensional similarity calculation: For pairs of entities of the same type to be matched, multi-dimensional similarity is calculated based on threat intelligence characteristics. The core calculation dimensions include: Core identifier similarity: such as the character consistency of IP / domain names (e.g., the similarity of characters after denormalization between 8.8.8.8 and 08.08.08.08), and the semantic similarity of attacker organization codes (e.g., the industry semantic relevance between APT28 and Fancy Bear). Such as the source of intelligence associated with an entity, the time of its historical appearance, and the overlap ratio of the entity with other sets of entities (e.g., entity A is associated with 3 sources, entity B is associated with 2 sources, and 1 of the sources overlaps, then the source overlap is 1 / 3≈33%). A weighted summation method is used to integrate multi-dimensional similarity into a comprehensive similarity (e.g., core identifier similarity accounts for 60% and attribute overlap accounts for 40%). Step 3: Duplicate Detection and Merging: Set a duplicate detection threshold. If the overall similarity is ≥80%, the entity is considered a duplicate. If the overall similarity of an entity pair exceeds the threshold, they are merged into one node. The valid attributes of each entity are retained (e.g., the intelligence source of the merged node includes all sources of the two entities, and the credibility is the weighted average of the credibility of each entity). Delete the original records of duplicate nodes and update the edges in the graph associated with duplicate nodes (e.g., adjust the edges that originally pointed to duplicate nodes to point to the merged nodes) to ensure the integrity of the graph relationships.

[0063] In a specific example, the initial attack graph contains two attacker organization nodes: Node 1 is labeled APT28, originating from threat intelligence platform A and associated with the attack tool Cobalt Strike; Node 2 is labeled Fancy Bear, originating from a report by security vendor B and also associated with the attack tool Cobalt Strike. Similarity calculations show that their core identifier semantic similarity is 100% (an industry-recognized figure), attribute overlap (associated tools, target industry) is 85%, and overall similarity reaches 94%. Exceeding this threshold, they are merged into a single APT28 / Fancy Bear node, retaining both source information and eliminating redundancy.

[0064] D3. The contradiction identification unit uses a combination of multi-source cross-validation and rule matching to identify contradictions. The specific processing flow is as follows: Step 1: Focus on core threat intelligence conflict scenarios and set two detection dimensions: Entity attribute conflicts: such as conflicts in the type attribute (malicious / legitimate) of the same IP address, conflicts in the severity level (high / medium) of the same vulnerability, and conflicts in the target system (Windows / Linux) of the same malicious file; Entity relationship conflict: such as the direct relationship contradiction between entity A belonging to entity B and entity A not belonging to entity B, or the behavioral relationship contradiction between entity A attacking entity C and entity A not attacking entity C; Step 2: For the same attribute / relationship of the same entity, collect statements from different intelligence sources. If there are two or more completely contradictory statements (e.g., Source 1 says the IP is malicious, while Source 2 says the IP is legitimate), it is preliminarily determined to be a conflict. In addition, it pre-defines industry-standard logical rules (such as malicious C2 servers and legitimate CDN nodes being mutually exclusive and unable to coexist). If an entity's attributes / relationships violate these rules, it is considered a conflict. Step 3: Add a conflict-to-resolve tag to the identified contradictory information, and record the conflict content in detail (such as the type attribute of IP1: malicious C2 server (source A) vs legitimate CDN node (source B)), the source of the conflict and the basic weight of each source, so as to provide a basis for the credibility calibration of the subsequent weight calculation unit.

[0065] In some embodiments, rule-based conflict detection may be used, or automated methods based on logical reasoning may be used for conflict identification.

[0066] D4. The weight calculation unit uses the Bayesian trust model as its core, combined with multiple influencing factors to calculate credibility. The specific processing flow is as follows: Step 1: Identify the three core influencing factors and their respective weights: Based on industry knowledge and historical performance, a basic reputation weight is set for different sources (e.g., a basic weight of 0.7-0.9 for authoritative threat intelligence platforms and a basic weight of 0.3-0.5 for ordinary open source communities). The more times an entity / relationship appears in different sources, the higher the credibility score (e.g., if the same IP is marked as malicious in 3 independent authoritative sources, the frequency score is higher than if it appears in only 1 source). A time decay coefficient is set. The more recent the intelligence extraction time (e.g., within the last 7 days), the higher the freshness score; the longer the time (e.g., more than 90 days), the lower the freshness score. Step 2: Based on the Bayesian trust model, integrate the scores of each influencing factor into the final credibility weight (value range 0-1, the closer to 1, the higher the credibility): Single-source intelligence: Credibility = Source credibility × Time freshness; Multi-source intelligence: Credibility = (Σ source credibility of each source × time freshness) / number of sources × frequency bonus coefficient (e.g., if 3 sources appear, the frequency bonus coefficient is 1.2); Step 3: Store the calculated final credibility weight as an attribute in the attribute table of the corresponding entity and relationship (e.g., the credibility of IP1's malicious attribute is 0.625, and the credibility of IP1's relationship to the APT organization is 0.58), supporting subsequent dynamic updates.

[0067] In some embodiments, weight calculation may also incorporate machine learning models to predict credibility.

[0068] The graph governance module ultimately outputs an optimized knowledge graph, which has the following characteristics: no isolated invalid nodes, no duplicate or redundant entities, traceable conflict information, and quantifiable credibility. This optimized knowledge graph can be directly passed to the graph evolution module for dynamic updates and iterations, providing high-quality core threat intelligence data for downstream output modules.

[0069] E. Knowledge graph evolution module, used for dynamic updates of the optimized knowledge graph. The updates include at least a time decay mechanism that reduces the credibility of outdated intelligence over time, and a feedback update mechanism that adjusts the graph based on external verification results to obtain a credible intelligence subgraph.

[0070] Furthermore, considering the time-sensitive nature of intelligence, differentiated attenuation rules (attenuation period, daily attenuation factor, minimum credibility threshold) are preset according to intelligence type (such as malicious IP / domain, vulnerability), and attenuation calculation is triggered periodically:

[0071] in, This represents the confidence level after decay (i.e., the final confidence level after an interval of t days, with a value ranging from 0 to 1). This represents the current credibility (i.e., the initial credibility before attenuation calculation, derived from the optimized value after graph governance). This is the attenuation factor (a daily attenuation ratio preset according to the intelligence type, such as setting the k value of malicious IPs to 0.004).

[0072] The decayed credibility of intelligence is obtained through the above formula; if the decayed credibility is lower than the threshold, it is marked for review or temporarily stored in the historical database to avoid outdated intelligence interfering with decision-making.

[0073] In some embodiments, the matching verification results are dynamically adjusted according to rules based on external verification results (automatic detection of SOC / SIEM, manual review, and third-party confirmation of CERT). Positive verification (confirmation of authenticity) increases credibility, negative verification (confirmation of falsehood) decreases credibility, and contradictory verification (partially true) is calculated comprehensively. Entity relationships are added or deleted (if a relationship is confirmed to be false, it is deleted; if a new association is added, it is added), and update logs are recorded for traceability.

[0074] In some embodiments, the module can also determine the evolution trend of intelligence, provide data support for adjusting time decay rules and identifying high-priority intelligence, and realize the extension from passive evolution to active prediction.

[0075] Specifically, evolutionary data from the graph over the past 1-3 months can be extracted, including the daily number of new intelligence reports, the average credibility of each type of intelligence report, and the number of new relationships with key entities (such as APT organizations). Time series models are constructed using either ARIMA (Autoregressive Integral Moving Average) or LSTM (Long Short-Term Memory) networks to adapt to different trend characteristics: The ARIMA model analyzes linear trend data such as the frequency of malicious IP occurrences and the frequency of vulnerability intelligence updates to predict intelligence changes over the next 1-2 weeks. LSTM models analyze non-linear trend data such as the evolution of APT group attack tactics and the growth of multi-entity relationships to identify potential threat hotspots (e.g., a surge in the relationships of a certain APT group may predict that it may launch a concentrated attack). Trend analysis results will be pushed to the security analysis platform in the form of trend curves and early warning prompts. For example, if it is predicted that the number of new ransomware tool intelligence will increase by 20% in the next 7 days, it is recommended to pay close attention.

[0076] In summary, the graph evolution module ultimately outputs a trusted intelligence subgraph, characterized by its credibility dynamically adjusted over time and with verification, its content undergoing closed-loop correction, its alignment with actual threats, and its direct availability for use by the output module.

[0077] F. Output module, used to output the processed trusted intelligence subgraph, providing interface calls, result display and early warning.

[0078] The output module provides standardized interfaces for downstream security devices such as firewalls, intrusion detection systems (IDS), and endpoint detection and response (EDR), as well as platforms such as security operations centers (SOC) and security information and incident management (SIEM). This enables automated invocation of trusted intelligence subgraphs, avoiding the inefficiency and error-prone nature of manual data entry. Specifically, it provides a RESTful API interface to support intelligence queries (such as querying related intelligence by malicious IP or APT organization type) and batch retrieval (such as retrieving high-confidence malicious intelligence newly added in the last 7 days), adapting to the common calling needs of most security devices. It also provides a GraphQL interface to support on-demand retrieval of entities and relationships (such as querying only the attack tools and victim industries associated with a specific APT organization), reducing data transmission volume, improving query efficiency, and adapting to scenarios requiring in-depth graph analysis (such as threat chain tracing on SOC platforms).

[0079] The output module transforms entities and relationships in the trusted intelligence subgraph into intuitive graphical and tabular information through a visual interface and exportable reports. This helps security analysts quickly understand threat relationships (such as attack chains and threat organization networks), while also supporting offline storage and reporting of intelligence. For example, an interactive interface is built using graph visualization frameworks (Cytoscape.js, D3.js). This interface displays the complete attack path in the form of a relationship chain between attackers, attack tools, malicious IPs, and victim companies, supporting node dragging, zooming, and clicking to view details (e.g., clicking on a malicious IP node displays its credibility, source, and associated file hash). It also shows the relationships between different APT organizations (such as sharing attack tools or coordinating attacks on the same industry), using different colors to indicate the risk level of organizations and assisting in identifying the size of threat groups.

[0080] In some embodiments, by combining actual enterprise asset information, new high-risk intelligence in the trusted intelligence subgraph is monitored in real time. When the intelligence matches enterprise assets (such as internal network IPs and business domain names), an alert is triggered and pushed to relevant personnel, realizing intelligence discovery, alerting, and response, and reducing attack surface exposure time. The specific process may include: the output module regularly synchronizing the enterprise asset list (such as server IPs, office terminal MAC addresses, and business domain names); when new high-confidence intelligence (such as malicious IPs or attack intelligence targeting a vulnerability) is added to the trusted intelligence subgraph, the association between the intelligence and the assets is automatically compared (such as whether the malicious IP is an IP of an enterprise partner or whether the vulnerability exists on the enterprise server); a confidence threshold is set (such as malicious intelligence confidence ≥ 60%), and an alert is triggered when the intelligence matching the asset meets the threshold; the alert information is pushed through multiple channels, including SIEM system alert pop-ups, security analyst workbench notifications, and internal enterprise security alert emails. The alert content includes intelligence details (such as malicious IPs and confidence levels), matching assets, and suggested response measures (such as temporarily blocking IPs or patching vulnerabilities).

[0081] This application provides a threat intelligence big data governance system based on knowledge graphs. The system includes a data acquisition module, an entity extraction module, a knowledge graph construction module, a graph governance module, a graph evolution module, and an output module. Through multi-module collaboration, this application first standardizes multi-source heterogeneous threat intelligence, then extracts entities and resolves ambiguities to construct a structured knowledge graph. After governance and optimization to eliminate redundancy and contradictions and calibrate credibility, time decay and feedback mechanisms ensure the timeliness and accuracy of the intelligence. Finally, multiple forms of output support downstream applications. This effectively solves the problems of single-item processing, lack of context, and difficulty in identifying redundant contradictions in existing technologies, significantly improving the credibility, completeness, and usability of threat intelligence, reducing false positive rates, and increasing data processing efficiency and the effectiveness of downstream security defenses.

[0082] Figure 2 This is a flowchart illustrating a knowledge graph-based threat intelligence big data governance method provided in an embodiment of this application. Figure 2 As shown, the method may include: Step S210: The data acquisition module collects multi-source heterogeneous raw threat intelligence, and performs preprocessing and standardization on the raw threat intelligence to obtain threat intelligence data. Step S220: The entity extraction module extracts entities and relationships between entities from the threat intelligence data, standardizes synonymous entities, and resolves different names for the same entity. Step S230: The knowledge graph construction module stores entities and relations in the graph database, defines the node and edge types of the graph, stores attribute information in the nodes and edges, and removes redundant nodes of the graph through entity alignment. Step S240: The graph governance module performs quality optimization and credibility calibration on the constructed threat intelligence knowledge graph to obtain an optimized knowledge graph. Step S250: The graph evolution module dynamically updates the optimized knowledge graph. The update includes at least a time decay mechanism that reduces the credibility of outdated intelligence over time, and a feedback update mechanism that adjusts the graph based on external verification results to obtain a credible intelligence subgraph. Step S260: The output module outputs the processed trusted intelligence subgraph, providing interface calls, result display, and early warning.

[0083] The steps of the knowledge graph-based threat intelligence big data governance method provided in the above embodiments of this application can be implemented by the modules of the above systems. Therefore, the beneficial effects of the knowledge graph-based threat intelligence big data governance method provided in the embodiments of this application will not be repeated here.

[0084] This application also provides an electronic device, such as... Figure 3 As shown, it includes a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340.

[0085] Memory 330 is used to store computer programs; When the processor 310 executes the program stored in the memory 330, it performs the following steps: The data acquisition module collects raw threat intelligence from multiple heterogeneous sources, and performs preprocessing and standardization on the raw threat intelligence to obtain threat intelligence data. The entity extraction module extracts entities and relationships between entities from the threat intelligence data, standardizes synonymous entities, and resolves different names for the same entity. The knowledge graph construction module stores entities and relationships in a graph database, defines the node and edge types of the graph, stores attribute information in nodes and edges, and removes redundant nodes in the graph through entity alignment. The knowledge graph governance module performs quality optimization and credibility calibration on the constructed threat intelligence knowledge graph to obtain an optimized knowledge graph; The graph evolution module dynamically updates the optimized knowledge graph, including at least a time decay mechanism that reduces the credibility of outdated intelligence over time, and a feedback update mechanism that adjusts the graph based on external verification results to obtain a credible intelligence subgraph. The output module outputs the processed trusted intelligence subgraph, providing interface calls, result display, and early warning.

[0086] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0087] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0088] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0089] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 2 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0091] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the knowledge graph-based threat intelligence big data governance methods described in the above embodiments.

[0092] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the knowledge graph-based threat intelligence big data governance methods described in the above embodiments.

[0093] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected," "coupled," or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0098] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the embodiments in this application are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments in this application.

[0099] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the embodiments of this application and their equivalents, then these modifications and variations are also intended to be included in the embodiments of this application.

Claims

1. A knowledge graph based threat intelligence big data governance system, characterized in that, The system comprises: a data acquisition module for acquiring multi-source heterogeneous original threat intelligence, and pre-processing and standardizing the original threat intelligence to obtain threat intelligence data; an entity extraction module for extracting entities and relationships between entities from the threat intelligence data, standardizing synonymous entities, and resolving the same entity with different names; a knowledge graph construction module for storing entities and relationships into a graph database, defining node types and edge types of the graph, storing attribute information in nodes and edges, and removing redundant nodes of the graph through entity alignment; a graph management module for quality optimization and credibility calibration of the constructed threat intelligence knowledge graph to obtain an optimized knowledge graph; a graph evolution module for dynamic updating of the optimized knowledge graph, including at least a time decay mechanism for reducing the credibility of outdated intelligence over time, and a feedback updating mechanism for adjusting the graph according to external verification results to obtain a credible intelligence subgraph; an output module for outputting the managed credible intelligence subgraph, providing interface calling, result display and early warning.

2. The system of claim 1, wherein, The graph management module comprises an isolated node detection unit for identifying isolated nodes, a redundancy merging unit for merging duplicate entities, a contradiction identification unit for discovering conflicting information, and a weight calculation unit for calculating the credibility of entities and relationships.

3. The system of claim 1, wherein, The data acquisition module is specifically configured to extract potential indicators in the original intelligence through regular expressions and natural language processing techniques, including domain names, IP addresses, file hashes, and vulnerability numbers; and the standardization includes converting the pre-processed intelligence into a unified format.

4. The system of claim 1, wherein, The entity extraction module extracts entities through a deep learning named entity recognition model, including malicious IPs, malicious domain names, malicious samples, attacker organizations and attack tools; and extracts attack path relationships between entities through dependency syntax analysis and graph neural networks.

5. The system of claim 1, wherein, The entity extraction module resolves the same entity by merging different names of the same attacker organization into the same entity.

6. The system of claim 1, wherein, The attribute information stored by the knowledge graph construction module includes timestamp, intelligence source weight and initial credibility; the entity alignment is achieved based on an attribute similarity algorithm, and the calculation dimensions of the attribute similarity include entity name, associated IP / domain name, historical attack record and source overlap.

7. The system of claim 2, wherein, The isolated node detection unit of the graph management module identifies isolated nodes through connected subgraph analysis and marks the isolated nodes as low-value or false intelligence candidates; and the redundancy merging unit aggregates duplicate entity nodes through an algorithm based on similarity matching. 8.A knowledge graph based threat intelligence big data governance method, characterized in that, The method comprises: a data acquisition module for acquiring multi-source heterogeneous original threat intelligence, and pre-processing and standardizing the original threat intelligence to obtain threat intelligence data; an entity extraction module for extracting entities and relationships between entities from the threat intelligence data, standardizing synonymous entities, and resolving the same entity with different names; The knowledge graph construction module stores entities and relations into a graph database, defines node types and edge types of the graph, stores attribute information in nodes and edges, and removes redundant nodes of the graph through entity alignment; The graph management module optimizes the quality and calibrates the credibility of the constructed threat intelligence knowledge graph, to obtain an optimized knowledge graph; The graph evolution module dynamically updates the optimized knowledge graph, and the update includes at least a time decay mechanism for reducing the credibility of outdated intelligence over time, and a feedback update mechanism for adjusting the graph according to external verification results, to obtain a credible intelligence subgraph; The output module outputs the managed credible intelligence subgraph, and provides interface calling, result display and early warning.

9. An electronic device, comprising: The electronic device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store a computer program; The processor is used to execute the program stored on the memory, and realize the method steps of claim 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of claim 8.

Images