A complete knowledge graph construction method based on multi-source vulnerability data

Abstract

The present invention discloses a method for constructing a complete knowledge graph based on multi-source vulnerability data, which mainly solves the problems of complex vulnerability data sources and missing relationships in the vulnerability database. The method includes: 1) collecting multi-source vulnerability data, where the data comes from unstructured vulnerability data recorded in CVE, CWE, CAPEC, and the security community; 2) normalizing sentences describing unstructured vulnerability information, performing sentence boundary detection and noun phrase normalization; 3) labeling the processed sentences with semantic roles, extracting sentence role construction data triples, and generating a vulnerability knowledge graph; 4) using node2vec to represent the nodes and relationships in the vulnerability knowledge graph into a low-dimensional dense vector space, and performing similarity calculation on the graph embedding results; 6) completing missing relationships through link prediction to obtain a complete knowledge graph. The present invention can establish a complete vulnerability database and effectively improve vulnerability retrieval efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and further relates to a method for constructing a complete knowledge graph based on multi-source vulnerability data, which can be used for vulnerability data management and analysis. Background Art

[0002] As cybersecurity becomes increasingly complex, attacks exploiting network vulnerabilities are becoming more frequent. In 2021, attackers exploited a buffer overflow vulnerability (CVE-2020-11292) to control the modem and dynamically update it from the application processor. This vulnerability allowed attackers to obtain text messages and call logs from mobile phone users, eavesdrop on conversations, and even remotely unlock SIM cards. For existing network security issues, collecting and analyzing vulnerability information to prevent future attacks is crucial.

[0003] A cybersecurity vulnerability refers to a flaw in the specific implementation of hardware, software, or protocols, or in system security policies, that could allow an attacker to gain unauthorized access or compromise a system. Currently, there are many widely used vulnerability datasets and collection platforms in the cybersecurity field, such as NVD, CNNVD, and WooYun. The National Vulnerability Database (NVD), a dataset managed and supported by the US government, includes CVE, CWE, and CVSS. CVE is a common name given to information security vulnerabilities. Using a unified naming format, CVE helps users share data across independent vulnerability databases. CWE is a community-developed list of common software and hardware security weaknesses. CWE is a formal checklist or dictionary for the development and security practitioner communities, listing common software and hardware weaknesses that may occur in architecture, design, code, or implementation. CVSS, or the Common Vulnerability Scoring System, aims to help people establish a standard for measuring vulnerability severity, allowing comparisons and prioritization of vulnerability resolution. In addition to the above vulnerability datasets, unstructured vulnerability data is also recorded in security communities and technical reports. Faced with the above multi-source vulnerability datasets, it is difficult to effectively and uniformly manage vulnerability data. Natural language processing technology is used to process and analyze the vulnerability information in technical reports, extract entities as new knowledge and add them to the vulnerability dataset, and build a unified vulnerability database to integrate data for efficient management.

[0004] Software vulnerabilities are often interconnected. For example, integer overflow is a specific case of improper software input validation. It may be caused by developers misunderstanding the numerical calculations of the programming language, resulting in incorrect conversions between numeric types. The occurrence of this vulnerability may further lead to inappropriate restrictions on operations within the memory buffer. The potential connections between these vulnerabilities are missing in existing vulnerability databases, making it difficult for security experts to quickly find other vulnerabilities associated with the current specified vulnerability through relationship retrieval. Regarding graph reasoning technology, symbolic representation-based reasoning was proposed early on. Common examples include symbolic reasoning based on Datalog and reasoning based on production rules. Symbolic representation-based reasoning is consistent with human logical thinking, but such methods have poor generalization capabilities.

[0005] In a patent document entitled "A Method and System for Constructing a Network Security Knowledge Graph" filed and disclosed on September 13, 2022, a method for constructing a network security knowledge graph was disclosed, including: constructing an ontology model in the network security field, extracting knowledge from structured data, semi-structured data, and unstructured text data to obtain network security entities, attributes, and relationships; Jia Yan disclosed a practical method for constructing a network security knowledge graph in "Engineering" in 2018, and in the reasoning part, a path sorting algorithm was used to infer new relationships. However, for multi-source vulnerability data sets, the reasoning efficiency of the path sorting algorithm is low and it is not suitable for large-scale vulnerability data sets. Summary of the Invention

[0006] The purpose of the invention is to overcome the defects of the above-mentioned existing technologies and propose a method for constructing a complete knowledge graph based on multi-source vulnerability data. It mainly solves the problems that the existing multi-source vulnerability data is difficult to effectively integrate, the vulnerability retrieval efficiency is low, and the constructed vulnerability library content is incomplete. The present invention can use knowledge graph technology to establish a complete vulnerability database and effectively improve the vulnerability retrieval efficiency.

[0007] The idea of ​​implementing the method of the present invention is as follows: first, structured and unstructured vulnerability data are obtained. The unstructured vulnerability data is tokenized to determine sentence boundaries to facilitate subsequent data extraction. The extracted data is preprocessed by normalization and standardization. Then, vulnerability triples are extracted from the existing data. Semantic role labeling (SRL) and a set of rules are used to extract the causal relationship of information flow and find the direction of edges, and triples are constructed from the output of SRL. Finally, node2vec is used to represent the nodes and relationships in the vulnerability knowledge graph into a low-dimensional dense vector space, and network node similarity is calculated to complete the missing relationships of the vulnerability data and obtain a complete knowledge graph.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] (1) Obtaining structured vulnerability data:

[0010] Data is obtained from the CVE vulnerability dataset, the CWE vulnerability dataset, and the CAPEC vulnerability dataset. This includes 250,000 vulnerability data items from CVE-1999 to CVE-2022, the CWE-699 series from CWE 4.9 Software Development, and CAPEC vulnerability data from the attack mechanism of the CAPEC Quick Access List. These three types of vulnerability data together constitute structured vulnerability data.

[0011] (2) Obtain unstructured vulnerability data and preprocess it:

[0012] (2a) Sentences describing unstructured vulnerability information are obtained from security communities and technical reports as unstructured vulnerability data, and boundary detection is performed on them. That is, conventional punctuation marks, bullet points, enumeration numbers, and titles are used as sentence delimiters to determine boundaries and implement sentence tokenization.

[0013] (2b) Mapping the different noun phrases and synonyms of verbs that appear in the tokenized sentences into the constructed dictionary, and determining different verbs with the same mapping results as representing the same action and normalizing them;

[0014] (2c) Using part-of-speech tagging and dependency syntax analysis, passive voice sentences are detected and whether the subject is missing. If missing, the subject is hidden, otherwise it is displayed. For sentences with displayed subjects, the subject and object positions are directly exchanged. For sentences with hidden subjects, the missing subject is analyzed according to the context, and the sentence is completed before the sentence is converted. The unstructured vulnerability data after preprocessing is obtained.

[0015] (3) Extract vulnerability data triples:

[0016] Perform triple standardization on the structured vulnerability data and the pre-processed unstructured vulnerability data, and organize them into the form of head entity, relationship, and tail entity, i.e. triples<head,relation,tail> Among them, semantic role tagging (SRL) is used to extract one verb node and two noun nodes from each tokenized sentence in the pre-processed unstructured vulnerability data, and causality is used to judge the sender and receiver of the two noun nodes to determine the head entity and the tail entity.

[0017] (4) Constructing ontology based on vulnerability data:

[0018] After preprocessing, select vulnerability data of the same type as the CVE, CWE, and CAPEC vulnerability datasets, and use them together with the structured vulnerability data obtained in step (1) as the ontology. Then, filter the common attributes of the ontology, that is, select ID, Name, Description, and Related Weaknesses as common attribute fields and store them in the database, and delete the remaining fields.

[0019] (5) Constructing a complete knowledge graph:

[0020] (5a) The triple data extracted from structured vulnerability data and unstructured vulnerability information are stored in the Neo4j graph database to build an incomplete knowledge graph;

[0021] (5b) Use the node2vec model to represent the nodes and relationships in the vulnerability knowledge graph into a low-dimensional dense vector space, and represent the graph structure in vector form;

[0022] (5c) In the graph structure after vector representation, heuristic scores are defined based on the neighbors of a pair of nodes (u, v). Weights are set for each neighbor node according to the links between common neighbor nodes, and their weighted sum is used as the similarity of the node pair.

[0023] (5d) By analyzing the Description field of the vulnerability detailed description information, verify whether the inferred relationship actually exists. If it does, the missing relationship will be completed; if it does not exist, the relationship will be discarded; and a complete knowledge graph will be obtained.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, since the vulnerability dataset of the present invention links a series of vulnerability-related datasets, compared with other vulnerability datasets, this dataset includes CVE, CWE, CPE, and vulnerability information in network security technical reports, and the data volume is huge. The CVE dataset records 184,861 vulnerability data, and the current CWE dataset collects a total of 927 vulnerabilities. Each vulnerability is described in detail and its relationship with other vulnerabilities is given. Therefore, the vulnerability database constructed using multi-source vulnerability data is more complete. When searching, as many other vulnerabilities related to the current vulnerability as possible can be found.

[0026] Second, in today's complex network environment, a universal network security ontology has not yet been constructed. Different application scenarios require different ontologies. The ontology constructed by this invention facilitates the data integration of various vulnerability data sources such as CVE and CWE, making the resulting ontology more suitable for current multi-source vulnerability datasets.

[0027] Third, the reasoning results of the present invention can be verified by examples, and the detailed description information of the vulnerability is used to verify the logic and accuracy of the reasoning. If the result of the reasoning meets the verification conditions, the obtained reasoning result is accurate; otherwise, the reasoning result is discarded. After the reasoning verification, it is found that the missing relationship between the two nodes does exist, and the missing relationship between the vulnerability nodes can be completed, making the vulnerability database more complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the present invention;

[0029] Figure 2 It is the knowledge graph reasoning sub-flowchart in the present invention;

[0030] Figure 3 This is a schematic diagram of the vulnerability knowledge graph constructed in the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example 1: Reference Figure 1 The present invention proposes a method for constructing a complete knowledge graph based on multi-source vulnerability data, which specifically includes the following steps:

[0033] Step 1. Obtain structured vulnerability data:

[0034] Data is obtained from the CVE vulnerability dataset, the CWE vulnerability dataset, and the CAPEC vulnerability dataset. This includes 250,000 vulnerability data items from CVE-1999 to CVE-2022, the CWE-699 series from CWE 4.9 Software Development, and CAPEC vulnerability data from the attack mechanism of the CAPEC Quick Access List. These three types of vulnerability data together constitute structured vulnerability data.

[0035] Step 2. Obtain unstructured vulnerability data and preprocess it:

[0036] (2a) Sentences describing unstructured vulnerability information are obtained from security communities and technical reports as unstructured vulnerability data, and boundary detection is performed on them. That is, conventional punctuation marks as well as bullet points, enumeration numbers, and titles are used as sentence delimiters to determine boundaries and implement sentence tokenization.

[0037] The boundary detection of unstructured vulnerability data is achieved by designing a marker to divide long sequences into multiple short sequences. A sentence is determined to be one that meets at least one of the following conditions:

[0038] a) begins with a capitalized subject and contains the necessary components of a complete sentence, namely, a subject, a predicate, and an object;

[0039] b) Starts with a verb included in the system call dictionary, i.e. omitting the subject but including the predicate and object.

[0040] (2b) Mapping the different noun phrases and synonyms of verbs that appear in the tokenized sentences into the constructed dictionary, and determining different verbs with the same mapping results as representing the same action and normalizing them;

[0041] (2c) Using part-of-speech tagging and dependency syntax analysis, passive voice sentences are detected and whether the subject is missing. If missing, the subject is hidden, otherwise it is displayed. For sentences with displayed subjects, the subject and object positions are directly exchanged. For sentences with hidden subjects, the missing subject is analyzed according to the context, and the sentence is completed before the sentence is converted. The unstructured vulnerability data after preprocessing is obtained.

[0042] Step 3. Extract vulnerability data triples:

[0043] Perform triple standardization on the structured vulnerability data and the pre-processed unstructured vulnerability data, and organize them into the form of head entity, relationship, and tail entity, i.e. triples<head,relation,tail> ; Among them, semantic role tagging SRL is used to extract one verb node and two noun nodes for each tokenized sentence in the pre-processed unstructured vulnerability data, and causality is used to judge the issuer and the recipient of the two noun nodes to determine the head entity and the tail entity. The head entity and the tail entity here are determined in the following way: using more than 32 different regular expressions and a database containing application names to detect system entity names, and then analyzing each noun phrase through the system entity extractor module to extract role nouns; associating each component in the sentence with a semantic label, and distinguishing the issuer and the recipient of the two nouns, with the issuer as the head entity and the recipient as the tail entity.

[0044] Step 4. Build ontology based on vulnerability data:

[0045] After preprocessing, select vulnerability data of the same type as the CVE, CWE, and CAPEC vulnerability datasets from the unstructured vulnerability data. Combine this data with the structured vulnerability data obtained in step 1 as the ontology. Then, filter the common attributes of the ontology. Specifically, select ID, Name, Description, and Related Weaknesses as common attribute fields and store them in the database. Delete the remaining fields.

[0046] Step 5. Reference Figure 2 , construct a complete knowledge graph:

[0047] (5a) The triple data extracted from structured vulnerability data and unstructured vulnerability information are stored in the Neo4j graph database to build an incomplete knowledge graph;

[0048] (5b) Use the node2vec model to represent the nodes and relationships in the vulnerability knowledge graph into a low-dimensional dense vector space, and represent the graph structure in vector form;

[0049] (5c) In the graph structure after vector representation, heuristic scores are defined based on the neighbors of a pair of nodes (x, y). A weight is set for each neighbor node based on the link status of the common neighbor nodes, and the weighted sum is used as the similarity of the node pair. The calculation formula is as follows:

[0050]

[0051] Where N(u) is the set of nodes adjacent to node u, N(x) is the set of nodes adjacent to node x, and N(y) is the set of nodes adjacent to node y. A higher similarity A(x,y) indicates a greater closeness between nodes x and y. Conversely, a value of 0 indicates that nodes x and y are not close.

[0052] (5d) By analyzing the Description field of the vulnerability detailed description information, verify whether the inferred relationship actually exists. If it does, the missing relationship will be completed; if it does not exist, the relationship will be discarded; and a complete knowledge graph will be obtained.

[0053] To verify whether the inferred relationship actually exists, the similarity is compared with the pre-set intimacy threshold to predict the association between the node pairs. Figure 3 This embodiment takes the prediction of the association between CWE-918 and CWE-79 as an example: first, the similarity between the verification server-side request forgery CWE-918 and the cross-site scripting attack CWE-79 is calculated. If the obtained value is greater than the preset threshold, it is determined that there is an association relationship between the two vulnerability nodes. The obtained relationship is verified for accuracy using the vulnerability detailed description message. If the vulnerability detailed description message is satisfied, the relationship exists; otherwise, the relationship is discarded. Among them, CWE-79 refers to a malicious attacker inserting malicious script code into a web page. When a user browses the page, the script code embedded in the web page will be executed, thereby achieving the purpose of maliciously attacking the user; CWE-918 refers to the attacker being able to send malicious access to other websites from a vulnerable web application. This access vulnerability exists in post parameters or XML.

[0054] Example 2: The overall implementation steps of this embodiment are the same as those of Example 1. A specific example of its implementation process is now given and further described in detail as follows:

[0055] Step a. Get structured vulnerability data

[0056] Obtain the allitems.csv file from the official CVE website, which contains 250,000 vulnerability data items from CVE-1999 to CVE-2022. Each CVE entry includes multiple fields, such as the CVE-ID, description, references, phase, votes, and comments. Obtain the CWE-699 series from CWE 4.9, Software Development. This vulnerability dataset contains frequently used and occurring vulnerabilities in software development, enabling better identification of potential errors in specific areas of software applications and providing early warnings for developers and architects. CAPEC currently lists over 500 attack types, including authentication attacks, cross-domain request forgery (CRF), where phishing websites exploit emails and web pages to trick users into clicking on them, thereby obtaining user identity information; injection attacks, where carefully crafted input data is submitted to control or disrupt the target's behavior, such as parameter injection, code inclusion, code injection, and SQL command injection; and resource exhaustion attacks and flooding attacks, where clients continuously utilize resources, repeatedly requesting resources, and thus exclude legitimate users, thereby benefiting the attacker. CAPEC vulnerability data is obtained from the attack mechanisms in the CAPEC quick access list. This category organizes attack patterns into layers based on the mechanisms commonly used to exploit vulnerabilities.

[0057] Step b. Get unstructured vulnerability data

[0058] Standardization primarily involves tokenization and normalization, which perform sentence boundary detection and word normalization, respectively. Both steps utilize system-built dictionaries. The first, the system call verb synonym dictionary, contains verbs representing system calls (such as write and fork operations) and their corresponding synonyms. These synonyms represent verbs that are likely to appear in vulnerability reports and are likely to refer to system calls. The second dictionary, the vulnerability noun dictionary, contains nouns and noun phrases commonly used in vulnerability reports, as well as different textual representations of the same concept.

[0059] b1) For tokenization, correctly defining sentence boundaries is crucial, as vulnerability information extraction operates at the sentence level. However, existing sentence tokenization methods typically only use classic punctuation marks (such as ".,!?") to determine sentence boundaries, making them inefficient when applied to vulnerability description text. To improve the efficiency of finding sentence boundaries, this paper designs a tokenizer specifically for vulnerability reports. In addition to commonly used sentence delimiters, this tokenizer uses bullet points, enumeration numbers, and titles as sentence delimiters, segmenting long sequences into multiple shorter sequences. After segmentation, each short sequence is considered a sentence if it meets one of the following conditions: 1) The sequence begins with a capitalized subject and contains the necessary components for a complete sentence, such as a subject, predicate, and object; 2) The sentence begins with a verb included in the system call dictionary, i.e., omitting the subject, and contains all the other components necessary to form a complete sentence after omitting the subject. The result of tokenization is a set of shorter sentences that are more likely to describe a single action, facilitating subsequent knowledge extraction.

[0060] b2) As for normalization, vulnerability reports usually contain some structural forms and synonyms, which will bring ambiguity and affect the quality of the final results. For example, C2 and C&C are different representations of the same entity, and "storage" and "save" represent the same action for system call operations. Normalization is the process of replacing multiple text representations of the same concept with the same text representation. The present invention uses two specially constructed dictionaries to normalize noun phrases and verbs. They map different nouns and synonyms of verbs that appear in vulnerability reports to entities and actions observed in logs. C2, C&C, and Command and Control are all mapped to a wildcard IP representing an IP address: .*. Normalization significantly reduces the ambiguity of the text. The verbs selected in the dictionary to represent synonyms, or the normalized noun representations, will act as entities in the subsequent extraction process.

[0061] b3) As the final step of standardization, converting passive sentences into active sentences helps discover the subject and object of the sentence. To perform this conversion, part-of-speech tagging (POS) and dependency parsing (DP) are used to detect passive sentences. DP specifies the grammatical connections and dependencies between words in a sentence, typically using arrows to represent dependencies. Sometimes, the arrows also indicate the specific grammatical relationship, such as whether it is a subject or object. Passive sentences are primarily represented by specific patterns in the dependency tree. For example, in the sentence "The downloaded file has been deleted by malware," in the dependency tree analysis, "be" is the marker of the passive sentence, "delete" as a verb is the parent node of the dependency tree, "downloaded file" is a noun phrase serving as the passive voice subject, and "malware" is an object. In some cases, the subject does not appear in a passive voice, but is hidden. By using part-of-speech tagging and dependency syntax analysis, we can detect the sentence structure and distinguish whether the passive voice has a displayed subject or a hidden subject. In the former case, we only need to exchange the positions of the subject and object and convert the passive verb into an active verb. In the latter case, we need to analyze the missing subject according to the context, complete the sentence, and then convert the sentence structure.

[0062] Step c. Extract vulnerability data triples

[0063] After the unstructured vulnerability data is normalized, the generated text appears in a standard form, where the subject (e.g., process), object (e.g., file, socket), and action predicate (e.g., exec) are obvious, ordered, and most redundant text is eliminated in the sentence. Although the text in this step is very simple, extracting subjects and objects as nodes and verbs as edges to create a graph often results in a fuzzy graph because some roles and relationships between subjects and objects can be expressed in the same sentence. To solve this problem, the present invention uses semantic role labeling (SRL) and a set of rules to extract the causal relationship of information flow and find the direction of the edge.

[0064] SRL discovers semantic roles in sentences and extracts two role nouns from each standardized sentence to identify which noun is the issuer of the action and which noun is the recipient of the action. SRL's role labeling is an act of correctly associating each component in a sentence with a semantic label. Nodes representing system entities are extracted from the roles generated by SRL, and useless parts of the roles that cannot constitute system entities are deleted. The present invention adopts a system entity extractor module that uses more than 32 different regular expressions and a database containing application names to detect possible system entity names (e.g., files, process names, IPs, registry keys). The system entity extractor module analyzes each noun phrase and deletes the entire phrase or the part that does not match the regular expression or application name.

[0065] We construct triples from the SRL output. For each sentence, it contains at least three roles, including a verb (a system call connecting nodes) and two nodes. We use causal inference to determine the correct direction of the edges in the graph, from the action issuer to the action receiver, to represent the causal relationship between the nodes. Finally, we extract the <node-edge-node> triple.

[0066] Step d. Construct an ontology for structured vulnerability data:

[0067] The cybersecurity knowledge base uses ontology as the foundation for knowledge representation. Ontologies integrate heterogeneous data and knowledge models from diverse vulnerability datasets, define domain-specific concepts, attributes, and relationships, facilitate data sharing and integration across diverse formats and standards, and support the inference of new information from existing information. Building on the UCO cybersecurity ontology, we integrated vulnerability data from CVE, CWE, and CAPEC, as well as vulnerability information from cybersecurity technical reports. We then modeled the vulnerability information extracted from these CVE, CWE, and CAPEC vulnerability databases and technical reports using ontology. First, we filtered CVE, CWE, and CAPEC attributes, selecting ID, Name, Description, and Related Weaknesses as common attribute fields and storing them in the database to unify the data representation format. We then extracted the content corresponding to ID, Name, Description, and Related Weaknesses and organized them into a key-value format, providing a unified template for knowledge graph construction.

[0068] Step e. Graph construction and reasoning

[0069] e1) The triples extracted from the unstructured vulnerability information are stored in the graph database Neo4j. The CSV file containing the structured vulnerability dataset is imported into Neo4j to construct a vulnerability knowledge base. Even with the constructed knowledge graph, there are still issues with incomplete content. For example, vulnerability X can create the necessary conditions to cause vulnerability Y to become vulnerable. When this occurs, the vulnerability dataset refers to X as the "primary node" of Y, and Y as the "result" of X. However, this relationship does not exist in the existing vulnerability data, necessitating knowledge graph reasoning to complete the missing relationship.

[0070] e2) A typical knowledge graph consists of millions of entity-relationship-entity triplets (h, r, t), where h, r, and t represent the head, relation, and tail of the triple, respectively. Given r and t, predicting h, or given h and r, predicting t, is denoted as (?, r, t) for the former and (h, r, ?) for the latter. For example, in a vulnerability dataset, (?, canprecede, cwe-119) predicts vulnerabilities that may occur before cwe-119, while (cwe-190, childof, ?) predicts vulnerabilities that have a childof relationship with cwe-190. This is essentially a knowledge graph completion task. Node2vec is used to represent the nodes and relations in the vulnerability knowledge graph into a low-dimensional dense vector space. The resulting graph is then embedded for similarity calculation to complete the downstream task of link prediction. Given all triples in the knowledge graph, node2vec learns a low-dimensional representation vector for each entity and relation, which preserves the structural information of the original knowledge graph. This method uses the embedding results of node2vec as the input of the link prediction model to complete the missing relationships in the vulnerability database, infer the missing relationships between vulnerabilities, build a complete vulnerability database, and complete the reasoning of potential relationships between vulnerabilities to complete the vulnerability database.

[0071] e3) Heuristic algorithms for link prediction all assume that more "similar" nodes are more likely to form links. This similarity is artificially defined; for example, it can be defined as the more common neighbors a node has, the more similar it is. Node2vec defines heuristic scores based on the neighbors of a node pair (u, v).

[0072] Traditional link prediction methods primarily analyze network topology features and calculate node similarities, assuming that nodes with high similarity will form links. Subsequently, a network node similarity measurement method was proposed. This method assigns weights to each neighboring node based on the links between their common neighbors and uses the weighted sum as the similarity between the node pairs. Based on the similarity scores, a threshold is set to predict the association between the node pairs.

[0073] e4) Explain the reasoning process for a specific missing relationship in vulnerability data: CWE-918 [Server-Side Request Forgery] → CWE-79 [Cross-Site Scripting]. This reasoning example is based on a vulnerability knowledge graph. The first step in reasoning a CWE chain is to use Neo4j's built-in tool, NEuler, and the node2vec algorithm to represent vulnerability information as a vector. The vulnerability knowledge graph is stored in Neo4j, a graph database. Using Neo4j's query language, Cypher, queries CWE-918 and CWE-79 are used to obtain accurate vulnerability descriptions. The second step is to calculate a similarity meter based on the links between common neighbor nodes. The result is compared with a threshold to determine whether the sample chain is a missing CWE chain. Finally, the vulnerability description is used to support the logic behind the sample chain and verify whether the links between CWE vulnerability data are missing relationships. For example, in the sample chain, CWE-918 is often referred to as SSRF (Server-Side Request Forgery), while CWE-79 is often referred to as XSS attack (exploiting vulnerabilities left in web page development). Generally speaking, attackers can use SSRF to forge requests and control response packets. In certain exploit scenarios, the content of the response packet is displayed on the webpage without filtering. If malicious JavaScript code is inserted into the response packet, the attacker will successfully construct an XSS attack. In the description, an attacker can execute an XSS (CWE-79) attack through Server-Side Request Forgery (CWE-918). Therefore, the logic behind the sample chain candidate is reasonable, and a reasonable CWE chain can be derived from the vulnerability knowledge graph reasoning, thus completing the missing reasonable relationship between CWE-918 and CWE-79.

[0074] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a complete knowledge graph based on multi-source vulnerability data, characterized by: include: (1) Obtaining structured vulnerability data: Data is obtained from the CVE vulnerability dataset, the CWE vulnerability dataset, and the CAPEC vulnerability dataset. This includes 250,000 vulnerability data items from CVE-1999 to CVE-2022, the CWE-699 series from CWE 4.9 Software Development, and CAPEC vulnerability data from the attack mechanism of the CAPEC Quick Access List. These three types of vulnerability data together constitute structured vulnerability data. (2) Obtain unstructured vulnerability data and preprocess it: (2a) Sentences describing unstructured vulnerability information are obtained from security communities and technical reports as unstructured vulnerability data, and boundary detection is performed on them. That is, conventional punctuation marks, bullet points, enumeration numbers, and titles are used as sentence delimiters to determine boundaries and implement sentence tokenization. (2b) Mapping the different noun phrases and synonyms of verbs that appear in the tokenized sentences into the constructed dictionary, and determining different verbs with the same mapping results as representing the same action and normalizing them; (2c) Using part-of-speech tagging and dependency parsing, the passive voice sentences are detected and whether the subject is missing. If missing, the subject is hidden, otherwise it is revealed. For sentences with revealed subjects, the sentence structure is directly transformed by exchanging the subject and object positions. For sentences with hidden subjects, the missing subject is analyzed based on the context, and the sentence is completed before the sentence structure is transformed. Obtain pre-processed unstructured vulnerability data; (3) Extract vulnerability data triples: Perform triple standardization on the structured vulnerability data and the pre-processed unstructured vulnerability data, and organize them into the form of head entity, relationship, and tail entity, i.e. triples<head,relation,tail> Among them, semantic role tagging (SRL) is used to extract one verb node and two noun nodes from each tokenized sentence in the pre-processed unstructured vulnerability data, and causality is used to judge the sender and receiver of the two noun nodes to determine the head entity and the tail entity. (4) Constructing ontology based on vulnerability data: After preprocessing, select vulnerability data of the same type as the CVE, CWE, and CAPEC vulnerability datasets, and use them together with the structured vulnerability data obtained in step (1) as the ontology. Then, filter the common attributes of the ontology, that is, select ID, Name, Description, and Related Weaknesses as common attribute fields and store them in the database, and delete the remaining fields. (5) Constructing a complete knowledge graph: (5a) The triple data extracted from structured vulnerability data and unstructured vulnerability information are stored in the Neo4j graph database to build an incomplete knowledge graph; (5b) Use the node2vec model to represent the nodes and relationships in the vulnerability knowledge graph into a low-dimensional dense vector space, and represent the graph structure in vector form; (5c) In the graph structure after vector representation, heuristic scores are defined based on the neighbors of a pair of nodes (x, y). Weights are set for each neighbor node based on the links of the common neighbor nodes, and their weighted sum is used as the similarity of the node pair. (5d) By analyzing the Description field of the vulnerability detailed description information, verify whether the inferred relationship actually exists. If it does, the missing relationship will be completed; if it does not exist, the relationship will be discarded; and a complete knowledge graph will be obtained.

2. The method according to claim 1, wherein: In step (2a), the boundary detection of the unstructured vulnerability data is performed by designing a marker to divide the long sequence into multiple short sequences, and determine that a sentence satisfies at least one of the following conditions: a) begins with a capitalized subject and contains the necessary components of a complete sentence, namely, a subject, a predicate, and an object; b) Starts with a verb included in the system call dictionary, i.e. omitting the subject but including the predicate and object.

3. The method according to claim 1, wherein: The head entity and tail entity in step (3) are determined as follows: using more than 32 different regular expressions and a database containing application names to detect system entity names, and then analyzing each noun phrase through the system entity extractor module to extract role nouns; associating each component in the sentence with a semantic label, and distinguishing the actor and the actor in the two nouns, and taking the actor as the head entity and the actor as the tail entity.

4. The method according to claim 1, wherein: The similarity calculation formula of the node pair in step (5c) is as follows: Among them, N(u) is the set of nodes adjacent to node u, N(x) is the set of nodes adjacent to node x, and N(y) is the set of nodes adjacent to node y.

5. The method according to claim 4, characterized in that: A higher value of the similarity A(x, y) indicates a greater intimacy between the two nodes x and y. Conversely, if it is 0, it directly indicates that the nodes x and y are not close.

6. The method according to claim 1, wherein: In the step (5d), whether the relationship obtained by reasoning actually exists is verified by comparing the similarity with a preset intimacy judgment threshold to predict the association between the node pairs, specifically as follows: first, the similarity between the verification server-side request forgery CWE-918 and the cross-site scripting attack CWE-79 is calculated. If the obtained value is greater than the preset threshold, it is determined that there is an association relationship between the two vulnerability nodes. The obtained relationship is verified for accuracy using the vulnerability detailed information. If the vulnerability detailed description message is met, the relationship exists, otherwise the relationship is discarded.

7. The method according to claim 6, characterized in that: CWE-79 refers to a vulnerability in which a malicious attacker inserts malicious script code into a web page. When a user browses the page, the script code embedded in the web page will be executed, thereby achieving the purpose of maliciously attacking the user. CWE-918 means that an attacker can send malicious access to other websites from a vulnerable web application. The access vulnerability exists in post parameters or XML.

Images