A dynamic threat graph generation method and system based on an improved support vector machine

Abstract

The application particularly relates to a dynamic threat graph generation method and system based on an improved support vector machine. The dynamic threat graph generation method and system based on the improved support vector machine intelligently extracts key knowledge information from text and machine data, constructs a dynamic threat graph generation engine based on an improved support vector machine algorithm, and outputs comprehensive and related intelligence graph data, so that the threat graph system actively identifies unknown attacks. The dynamic threat graph generation method and system based on the improved support vector machine convert an original rule-dependent mode into a rule-free threat graph generation engine, thereby reducing a false alarm rate of rule detection, improving existing threat detection efficiency, and effectively ensuring the stability of overall cloud center business.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a method and system for generating dynamic threat maps based on an improved support vector machine. Background Technology

[0002] With the continuous development of internet application technologies and the increasing prevalence of new threats and attacks, typified by Advanced Persistent Threats (APTs), a significant number of government and enterprise organizations directly connect to the internet to conduct a large amount of their work. However, the cybersecurity trend is becoming increasingly severe year by year, with frequent internet cyberattacks threatening current cyberspace security. To learn from past major cyberattacks, security professionals carefully analyze previous incidents and produce relevant reports.

[0003] On the other hand, threat graph technology has advantages in knowledge search, fusion, reasoning, and visualization, and has been successfully applied in multiple fields. In the current complex cyberspace security context, how to automatically and intelligently analyze numerous attacks to extract key fingerprints from reports, thereby constructing a threat intelligence knowledge graph, has become an important research topic in the field of information security.

[0004] To reduce the false alarm rate of rule detection and improve the detection efficiency of existing threats, this invention proposes a dynamic threat map generation method and system based on an improved support vector machine. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient method for generating dynamic threat maps based on an improved support vector machine.

[0006] This invention is achieved through the following technical solution:

[0007] A dynamic threat graph generation method based on an improved support vector machine is characterized by: intelligently extracting key knowledge information from text and machine data, constructing a dynamic threat graph generation engine based on the improved support vector machine algorithm, and producing comprehensive and related intelligence graph data, enabling the threat graph system to proactively identify unknown attacks.

[0008] Includes the following steps:

[0009] Step S1: Knowledge Catch

[0010] We read data from publicly available or semi-public text data on the Internet as raw corpus, classify and clean the read data, and finally perform semi-automated manual annotation on the cleaned intelligence data to build a preliminary intelligence corpus.

[0011] Step S2: Behavior Modeling

[0012] Threat knowledge is extracted and data is classified and statistically analyzed based on an intelligence corpus to obtain threat intelligence data; a threat knowledge prediction model is constructed and trained on the intelligence data using a support vector machine algorithm;

[0013] When extracting threat knowledge, events, risks, and threat knowledge are extracted sequentially along the time dimension; a threat knowledge prediction model is constructed based on the triples of event, risk, and threat knowledge.

[0014] Step S3: Generate a dynamic threat map.

[0015] Threat profiles are constructed based on the threat intelligence data obtained after behavioral entity modeling in step S2, and a threat graph is built.

[0016] In step S1, a whitelist mechanism is introduced during the cleaning process of the read data; the crawled and collected text data is filtered through the whitelist before being output for use.

[0017] In step S1, the publicly available or semi-public text data includes, but is not limited to, publicly available vulnerability databases, publicly available microblogs of hacker organizations, and publicly available microblog data of hacker organizations.

[0018] A whitelist of publicly available or semi-public text data must include at least the following dimensions:

[0019] 1) Vulnerability intelligence database whitelist dimensions: vulnerability number (vulNumber), vulnerability content (vulDesc), vulnerability level (vulLevel), vulnerability disclosure time (vulCommitDate), vulnerability update time (vulUpdate), and vulnerability submitter (vulCommitAuthor);

[0020] 2) Hacker organization whitelist dimensions: Weibo ID (AuthorID), Hacker organization ID (HackersID), content data published by the hacker organization (HackersPublishDesc), ID of subscribed users (SeedID), number of reads (ReadCount), content commented on by the hacker organization (HackersCommits), and user IDs followed by the hacker organization (HackersFocusUserID).

[0021] In step S1, the publicly disclosed vulnerability database includes, but is not limited to, the CVE vulnerability database and the SYNK vulnerability database.

[0022] For publicly available vulnerability databases, a web crawler is written to crawl vulnerabilities in XML format. The data structure includes six fields: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate, and vulnerability submitter vulCommitAuthor, resulting in a vulnerability set E = (vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor).

[0023] The hacker organization is labeled as v, and a set of hacker organizations P is constructed, where v∈P. Historical data that hacker organization v has ever published is crawled, as well as the Weibo user IDs that have liked, forwarded, and commented on each piece of data for each piece of intelligence, i.e., the Weibo user IDs that have clicked on the corresponding threat intelligence, which are used as subsequent threat graph data.

[0024] After whitelist filtering, an intelligence knowledge base set (E, P) is obtained, generating m. i m-dimensional vector i This refers to the current whitelist dimension.

[0025] In step S2, since the support vector machine algorithm has the problem of blind parameter selection during the training process when predicting the model, the particle swarm optimization algorithm is used to optimize the selected parameters when constructing the threat knowledge prediction model, thereby improving the accuracy of the prediction model.

[0026] The specific steps are as follows:

[0027] Step S2.1: Match the intelligence knowledge base with the built-in feature library using feature codes. If the match is successful, terminate the matching process and mark and store the matching result data. If all features in the feature library fail to match, terminate the matching process.

[0028] Step S2.2: Read the pre-made model training sample set, use the extracted event, risk and threat knowledge triples as initial training parameters to complete the first training of the support vector machine model, and obtain the initial prediction model.

[0029] The support vector machine model employs the subgradient descent algorithm, as shown in the following formula:

[0030]

[0031] Where m is the event set, and m is the current whitelist dimension. i x is the i-th element in the event set m; i For the i-th risk element, y i Let n be the number of iterations for the i-th threat knowledge;

[0032] Step S2.3: Continue reading the model training sample set and use the initial prediction model to complete the situation value prediction of the test sample to obtain the initial prediction result;

[0033] Step S2.4: Correct the prediction results using the particle swarm optimization algorithm;

[0034] The fitness evaluation function R of the particle swarm optimization algorithm is formulated as follows:

[0035]

[0036] Among them, f 1n For the initial prediction result, f 2n To test the true situation values ​​in the sample set;

[0037] The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set. When the fitness evaluation function R satisfies the entire convergence condition, the initial prediction model is the final prediction model. Otherwise, the particle swarm optimization algorithm iterates and passes the second set of training parameters to the support vector machine model. The support vector machine model rereads the training sample set for a second training to obtain the second-generation prediction model. The accuracy of the second-generation prediction model is then verified using the test sample set. This process continues until the final prediction model that satisfies the convergence condition of the fitness evaluation function R is obtained. The prediction result is then output.

[0038] In step S2, the knowledge construction process is scheduled by using a weighted round-robin method through the round-robin algorithm.

[0039] In step S3, taking the security event as the ontology, and through the abstraction of the entire attack process, using entity information including but not limited to attack domain names and IPs as the tuple, a threat intelligence ontology structure is created from top to bottom based on the characteristics and rules of the threat intelligence domain. Six threat information components are proposed for the threat intelligence ontology, namely, security event, threat subject, attack indicator, observable object, attack method, and attack target.

[0040] By using a multi-factor threat intelligence named entity recognition model and a graph neural network-based threat intelligence entity relation extraction model, entity and relation triples (events, risks, and threat knowledge) of threat intelligence texts are obtained. The obtained triples are stored in a graph database to generate a threat graph.

[0041] A dynamic threat graph generation system based on an improved support vector machine is characterized by comprising a knowledge learning module, a behavioral entity modeling module, a dynamic threat graph generation module, and a logical storage module.

[0042] The knowledge learning module reads data from publicly available or semi-public text data on the Internet as raw corpus, classifies and cleans the read data, and finally performs semi-automatic manual annotation on the cleaned intelligence data to build a preliminary intelligence corpus.

[0043] The behavioral entity modeling module includes a threat knowledge extraction submodule, a threat knowledge construction submodule, and a task scheduling submodule.

[0044] The threat knowledge extraction submodule performs threat knowledge extraction and data classification statistics based on the intelligence corpus, realizes data clustering analysis, and obtains threat intelligence data.

[0045] The threat knowledge construction submodule is responsible for building a threat knowledge prediction model and training the intelligence data using the support vector machine algorithm.

[0046] The task scheduling submodule uses a weighted round-robin algorithm to schedule the knowledge construction process.

[0047] When extracting threat knowledge, the threat knowledge extraction submodule extracts event, risk, and threat knowledge in a time-series manner; and constructs a threat knowledge prediction model based on the event, risk, and threat knowledge triples.

[0048] The dynamic threat graph generation module performs threat characterization on the threat intelligence data obtained after behavioral entity modeling and constructs a threat graph.

[0049] The logical storage module includes two storage modes: a distributed MySQL database cluster and a distributed HBase database. The distributed MySQL database cluster is used to store the sample set for model training and the results of attack determination. The distributed HBase database is used to store information related to signal capture volume.

[0050] The knowledge learning module is equipped with a whitelist mechanism, which filters the crawled and collected text data before outputting it for use.

[0051] The publicly available or semi-public text data includes, but is not limited to, publicly available vulnerability databases, publicly available microblogs of hacker organizations, and publicly available microblog data of hacker organizations.

[0052] A whitelist of publicly available or semi-public text data must include at least the following dimensions:

[0053] 1) Vulnerability intelligence database whitelist dimensions: vulnerability number (vulNumber), vulnerability content (vulDesc), vulnerability level (vulLevel), vulnerability disclosure time (vulCommitDate), vulnerability update time (vulUpdate), and vulnerability submitter (vulCommitAuthor);

[0054] 2) Hacker organization whitelist dimensions: Weibo ID (AuthorID), Hacker organization ID (HackersID), content data published by the hacker organization (HackersPublishDesc), ID of subscribed users (SeedID), number of reads (ReadCount), content commented on by the hacker organization (HackersCommits), and user IDs followed by the hacker organization (HackersFocusUserID);

[0055] Publicly available vulnerability databases include, but are not limited to, the CVE vulnerability database and the SYNK vulnerability database;

[0056] For publicly available vulnerability databases, a web crawler is written to crawl vulnerabilities in XML format. The data structure includes six fields: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate, and vulnerability submitter vulCommitAuthor, resulting in a vulnerability set E = (vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor).

[0057] The hacker organization is labeled as v, and a set of hacker organizations P is constructed, where v∈P. Historical data that hacker organization v has ever published is crawled, as well as the Weibo user IDs that have liked, forwarded, and commented on each piece of data for each piece of intelligence, i.e., the Weibo user IDs that have clicked on the corresponding threat intelligence, which are used as subsequent threat graph data.

[0058] After whitelist filtering, an intelligence knowledge base set (E, P) is obtained, generating m. i m-dimensional vector i This refers to the current whitelist dimension.

[0059] The threat knowledge construction submodule uses a particle swarm optimization algorithm to optimize the selected parameters, thereby improving the accuracy of the prediction model.

[0060] The specific steps are as follows:

[0061] Step S2.1: Match the intelligence knowledge base with the built-in feature library using feature codes. If the match is successful, terminate the matching process and mark and store the matching result data. If all features in the feature library fail to match, terminate the matching process.

[0062] Step S2.2: Read the pre-made model training sample set, use the extracted event, risk and threat knowledge triples as initial training parameters to complete the first training of the support vector machine model, and obtain the initial prediction model.

[0063] The support vector machine model employs the subgradient descent algorithm, as shown in the following formula:

[0064]

[0065] Where m is the event set, and m is the current whitelist dimension. i x is the i-th element in the event set m; i For the i-th risk element, y i Let n be the number of iterations for the i-th threat knowledge;

[0066] Step S2.3: Continue reading the model training sample set and use the initial prediction model to complete the situation value prediction of the test sample to obtain the initial prediction result;

[0067] Step S2.4: Correct the prediction results using the particle swarm optimization algorithm;

[0068] The fitness evaluation function R of the particle swarm optimization algorithm is formulated as follows:

[0069]

[0070] Among them, f 1n For the initial prediction result, f 2n To test the true situation values ​​in the sample set;

[0071] The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set. When the fitness evaluation function R satisfies the entire convergence condition, the initial prediction model is the final prediction model. Otherwise, the particle swarm optimization algorithm iterates and passes the second set of training parameters to the support vector machine model. The support vector machine model rereads the training sample set for a second training to obtain the second-generation prediction model. The accuracy of the second-generation prediction model is then verified using the test sample set. This process continues until the final prediction model that satisfies the convergence condition of the fitness evaluation function R is obtained. The prediction result is then output.

[0072] The dynamic threat graph generation module takes security events as its ontology. By abstracting the entire attack process, it uses entity information, including but not limited to attack domain names and IPs, as its primitives. Based on the characteristics and rules of the threat intelligence field, it creates a threat intelligence ontology structure from top to bottom and proposes six threat information components for the threat intelligence ontology: security events, threat subjects, attack indicators, observable objects, attack methods, and attack targets.

[0073] The beneficial effects of this invention are: the dynamic threat graph generation method and system based on improved support vector machine transforms the original method that relies on expert rules into a ruleless threat graph generation engine, thereby reducing the false alarm rate of rule detection, improving the detection efficiency of existing threats, and effectively ensuring the stability of the overall business of the cloud center. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Appendix Figure 1 This is a schematic diagram of the dynamic threat map generation method based on the improved support vector machine of the present invention.

[0076] Appendix Figure 2 This is a schematic diagram of the improved support vector machine algorithm of this invention. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0078] The improved Support Vector Machine (SVM) algorithm is a binary classification model that, after modification, can be used for multi-class classification problems. SVMs are divided into two main categories: linear and nonlinear kernels. The main idea is to find a hyperplane in the space that effectively divides all data samples, such that the distance from all data points in the sample set to this hyperplane is minimized.

[0079] The dynamic threat graph generation engine, based on expert experience, recognizes that the vast majority of attack intelligence data on the internet has complex interrelationships. For example, virus and Trojan samples typically employ encrypted remote command and control (C&C) instructions to remotely control infected machines (botnets infected with Trojans or viruses). The virus / Trojan sample signature information, C&C remote address, URL, IP address, communication domain name information, and the corresponding domain name's registrant, holder, contact information, phone number, registration status, and registration time are usually compiled into WHOIS data. This core and highly sensitive data is automatically output to a database after being processed by the dynamic threat graph generation engine, ultimately generating operational strategies to guide the construction of current cybersecurity defense systems.

[0080] Current threat graph generation models in the cybersecurity industry rely solely on a single-dimensional threat database to match information sources for target clients, such as URLs, compromised domains, and compromised IP data. This single-dimensional threat intelligence data does not correlate with threat intelligence data from various complex dimensions, resulting in a relatively low overall matching accuracy. When data distortion or abnormal matching errors occur, it greatly increases the likelihood of false positives on the user side. However, by combining an improved support vector machine algorithm with a dynamic threat graph generation engine, the generated intelligence data can effectively combine the strengths of various aspects, reduce the false positive rate, and effectively detect known attacks.

[0081] This dynamic threat graph generation method based on improved support vector machine intelligently extracts key knowledge information from text and machine data, and constructs a dynamic threat graph generation engine based on the improved support vector machine algorithm to produce comprehensive and related intelligence graph data, enabling the threat graph system to proactively identify unknown attacks.

[0082] Includes the following steps:

[0083] Step S1: Knowledge Catch

[0084] We read data from publicly available or semi-public text data on the Internet as raw corpus, classify and clean the read data, and finally perform semi-automated manual annotation on the cleaned intelligence data to build a preliminary intelligence corpus.

[0085] Step S2: Behavior Modeling

[0086] Threat knowledge is extracted and data is classified and statistically analyzed based on an intelligence corpus to obtain threat intelligence data; a threat knowledge prediction model is constructed and trained on the intelligence data using a support vector machine algorithm;

[0087] When extracting threat knowledge, events, risks, and threat knowledge are extracted sequentially along the time dimension; a threat knowledge prediction model is constructed based on the triples of event, risk, and threat knowledge.

[0088] Step S3: Generate a dynamic threat map.

[0089] Threat profiles are constructed based on the threat intelligence data obtained after behavioral entity modeling in step S2, and a threat graph is built.

[0090] It is particularly important to note that both publicly available vulnerability intelligence databases and hacker group Weibo data contain noisy data. In step S1, a whitelist mechanism is introduced during the data cleaning process; the crawled and collected text data is filtered through the whitelist before being output for use.

[0091] In step S1, the publicly available or semi-public text data includes, but is not limited to, publicly available vulnerability databases, publicly available microblogs of hacker organizations, and publicly available microblog data of hacker organizations.

[0092] A whitelist of publicly available or semi-public text data must include at least the following dimensions:

[0093] 1) Vulnerability intelligence database whitelist dimensions: vulnerability number (vulNumber), vulnerability content (vulDesc), vulnerability level (vulLevel), vulnerability disclosure time (vulCommitDate), vulnerability update time (vulUpdate), and vulnerability submitter (vulCommitAuthor);

[0094] 2) Hacker organization whitelist dimensions: Weibo ID (AuthorID), Hacker organization ID (HackersID), content data published by the hacker organization (HackersPublishDesc), ID of subscribed users (SeedID), number of reads (ReadCount), content commented on by the hacker organization (HackersCommits), and user IDs followed by the hacker organization (HackersFocusUserID).

[0095] In step S1, the publicly disclosed vulnerability database includes, but is not limited to, the CVE vulnerability database and the SYNK vulnerability database.

[0096] For publicly available vulnerability databases, a web crawler is written to crawl vulnerabilities in XML format. The data structure includes six fields: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate, and vulnerability submitter vulCommitAuthor, resulting in a vulnerability set E = (vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor).

[0097] The hacker organization is labeled as v, and a set of hacker organizations P is constructed, where v∈P. Historical data that hacker organization v has ever published is crawled, as well as the Weibo user IDs that have liked, forwarded, and commented on each piece of data for each piece of intelligence, i.e., the Weibo user IDs that have clicked on the corresponding threat intelligence, which are used as subsequent threat graph data.

[0098] After whitelist filtering, an intelligence knowledge base set (E, P) is obtained, generating m. i m-dimensional vector i This refers to the current whitelist dimension.

[0099] In step S2, since the support vector machine algorithm has the problem of blind parameter selection during the training process when predicting the model, the particle swarm optimization algorithm is used to optimize the selected parameters when constructing the threat knowledge prediction model, thereby improving the accuracy of the prediction model.

[0100] The specific steps are as follows:

[0101] Step S2.1: Match the intelligence knowledge base with the built-in feature library using feature codes. If the match is successful, terminate the matching process and mark and store the matching result data. If all features in the feature library fail to match, terminate the matching process.

[0102] Step S2.2: Read the pre-made model training sample set, use the extracted event, risk and threat knowledge triples as initial training parameters to complete the first training of the support vector machine model, and obtain the initial prediction model.

[0103] The support vector machine model employs the subgradient descent algorithm, as shown in the following formula:

[0104]

[0105] Where m is the event set, and m is the current whitelist dimension. i x is the i-th element in the event set m; i For the i-th risk element, y i Let n be the number of iterations for the i-th threat knowledge;

[0106] Step S2.3: Continue reading the model training sample set and use the initial prediction model to complete the situation value prediction of the test sample to obtain the initial prediction result;

[0107] Step S2.4: Correct the prediction results using the particle swarm optimization algorithm;

[0108] The fitness evaluation function R of the particle swarm optimization algorithm is formulated as follows:

[0109]

[0110] Among them, f 1n For the initial prediction result, f 2n To test the true situation values ​​in the sample set;

[0111] The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set. When the fitness evaluation function R satisfies the entire convergence condition, the initial prediction model is the final prediction model. Otherwise, the particle swarm optimization algorithm iterates and passes the second set of training parameters to the support vector machine model. The support vector machine model rereads the training sample set for a second training to obtain the second-generation prediction model. The accuracy of the second-generation prediction model is then verified using the test sample set. This process continues until the final prediction model that satisfies the convergence condition of the fitness evaluation function R is obtained. The prediction result is then output.

[0112] In step S2, the knowledge construction process is scheduled by using a weighted round-robin method through the round-robin algorithm.

[0113] In step S3, taking the security event as the ontology, and through the abstraction of the entire attack process, using entity information including but not limited to attack domain names and IPs as the tuple, a threat intelligence ontology structure is created from top to bottom based on the characteristics and rules of the threat intelligence domain. Six threat information components are proposed for the threat intelligence ontology, namely, security event, threat subject, attack indicator, observable object, attack method, and attack target.

[0114] By using a multi-factor threat intelligence named entity recognition model and a graph neural network-based threat intelligence entity relation extraction model, entity and relation triples (events, risks, and threat knowledge) of threat intelligence texts are obtained. The obtained triples are stored in a graph database to generate a threat graph.

[0115] The dynamic threat graph generation system based on the improved support vector machine includes a knowledge learning module, a behavioral entity modeling module, a dynamic threat graph generation module, and a logical storage module.

[0116] The knowledge learning module reads data from publicly available or semi-public text data on the Internet as raw corpus, classifies and cleans the read data, and finally performs semi-automatic manual annotation on the cleaned intelligence data to build a preliminary intelligence corpus.

[0117] The behavioral entity modeling module includes a threat knowledge extraction submodule, a threat knowledge construction submodule, and a task scheduling submodule.

[0118] The threat knowledge extraction submodule performs threat knowledge extraction and data classification statistics based on the intelligence corpus, realizes data clustering analysis, and obtains threat intelligence data.

[0119] The threat knowledge construction submodule is responsible for building a threat knowledge prediction model and training the intelligence data using the support vector machine algorithm.

[0120] The task scheduling submodule uses a weighted round-robin algorithm to schedule the knowledge construction process.

[0121] When extracting threat knowledge, the threat knowledge extraction submodule extracts event, risk, and threat knowledge in a time-series manner; and constructs a threat knowledge prediction model based on the event, risk, and threat knowledge triples.

[0122] The dynamic threat graph generation module performs threat characterization on the threat intelligence data obtained after behavioral entity modeling and constructs a threat graph.

[0123] The logical storage module includes two storage modes: a distributed MySQL database cluster and a distributed HBase database. The distributed MySQL database cluster is used to store the sample set for model training and the results of attack determination. The distributed HBase database is used to store information related to signal capture volume.

[0124] The knowledge learning module is equipped with a whitelist mechanism, which filters the crawled and collected text data before outputting it for use.

[0125] The publicly available or semi-public text data includes, but is not limited to, publicly available vulnerability databases, publicly available microblogs of hacker organizations, and publicly available microblog data of hacker organizations.

[0126] A whitelist of publicly available or semi-public text data must include at least the following dimensions:

[0127] 1) Vulnerability intelligence database whitelist dimensions: vulnerability number (vulNumber), vulnerability content (vulDesc), vulnerability level (vulLevel), vulnerability disclosure time (vulCommitDate), vulnerability update time (vulUpdate), and vulnerability submitter (vulCommitAuthor);

[0128] 2) Hacker organization whitelist dimensions: Weibo ID (AuthorID), Hacker organization ID (HackersID), content data published by the hacker organization (HackersPublishDesc), ID of subscribed users (SeedID), number of reads (ReadCount), content commented on by the hacker organization (HackersCommits), and user IDs followed by the hacker organization (HackersFocusUserID);

[0129] Publicly available vulnerability databases include, but are not limited to, the CVE (Common Vulnerability and Exposures) vulnerability database and the SYNK vulnerability database;

[0130] For publicly available vulnerability databases, a web crawler is written to perform categorized crawling in XML (Extensible Markup Language) format. The data structure includes six fields: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate, and vulnerability submitter vulCommitAuthor, resulting in a vulnerability set E = (vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor).

[0131] The hacker organization is labeled as v, and a set of hacker organizations P is constructed, where v∈P. Historical data that hacker organization v has ever published is crawled, as well as the Weibo user IDs that have liked, forwarded, and commented on each piece of data for each piece of intelligence, i.e., the Weibo user IDs that have clicked on the corresponding threat intelligence, which are used as subsequent threat graph data.

[0132] After whitelist filtering, an intelligence knowledge base set (E, P) is obtained, generating m. i m-dimensional vector i This refers to the current whitelist dimension.

[0133] The threat knowledge construction submodule uses a particle swarm optimization algorithm to optimize the selected parameters, thereby improving the accuracy of the prediction model.

[0134] The specific steps are as follows:

[0135] Step S2.1: Match the intelligence knowledge base with the built-in feature library using feature codes. If the match is successful, terminate the matching process and mark and store the matching result data. If all features in the feature library fail to match, terminate the matching process.

[0136] Step S2.2: Read the pre-made model training sample set, use the extracted event, risk and threat knowledge triples as initial training parameters to complete the first training of the support vector machine model, and obtain the initial prediction model.

[0137] The support vector machine model employs the subgradient descent algorithm, as shown in the following formula:

[0138]

[0139] Where m is the event set, and m is the current whitelist dimension. i x is the i-th element in the event set m; i For the i-th risk element, y i Let n be the number of iterations for the i-th threat knowledge;

[0140] Step S2.3: Continue reading the model training sample set and use the initial prediction model to complete the situation value prediction of the test sample to obtain the initial prediction result;

[0141] Step S2.4: Correct the prediction results using the particle swarm optimization algorithm;

[0142] The fitness evaluation function R of the particle swarm optimization algorithm is formulated as follows:

[0143]

[0144] Among them, f 1n For the initial prediction result, f 2n To test the true situation values ​​in the sample set;

[0145] The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set. When the fitness evaluation function R satisfies the entire convergence condition, the initial prediction model is the final prediction model. Otherwise, the particle swarm optimization algorithm iterates and passes the second set of training parameters to the support vector machine model. The support vector machine model rereads the training sample set for a second training to obtain the second-generation prediction model. The accuracy of the second-generation prediction model is then verified using the test sample set. This process continues until the final prediction model that satisfies the convergence condition of the fitness evaluation function R is obtained. The prediction result is then output.

[0146] The dynamic threat graph generation module takes security events as its ontology. By abstracting the entire attack process, it uses entity information, including but not limited to attack domain names and IPs, as its primitives. Based on the characteristics and rules of the threat intelligence field, it creates a threat intelligence ontology structure from top to bottom and proposes six threat information components for the threat intelligence ontology: security events, threat subjects, attack indicators, observable objects, attack methods, and attack targets.

[0147] Compared with existing technologies, this dynamic threat map generation method and system based on improved support vector machines has the following characteristics:

[0148] (1) Based on the improved support vector machine algorithm model, and the particle swarm algorithm is introduced to correct the model, the entire system can intelligently extract intelligence knowledge, thereby improving the overall detection rate of the threat map for unknown attacks.

[0149] (2) The unique dynamic threat map generation engine automatically associates intelligence knowledge and extracts unique and effective detection rules from the attack when encountering unknown attacks, especially 0-day attacks and -1-day attacks, thereby reducing the cost of manual intervention and improving detection efficiency.

[0150] (3) A distributed deployment method is adopted. Due to the complex environment of the current cloud center, each cloud center has its own business logic. Therefore, the detection rules for each cloud center will be different in nature, which reduces the overall false alarm rate of the system to a certain extent and effectively avoids the impact on the business.

[0151] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. An improved support vector machine based dynamic threat graph generation method, characterized in that: Intelligently extract key knowledge information from text and machine data, build a dynamic threat map generation engine based on an improved support vector machine algorithm, and output comprehensive and related intelligence map data to enable the threat map system to actively identify unknown attacks; The method comprises the following steps: Step S1, knowledge learning Read data from publicly available or semi-public text data on the Internet, classify and clean the read data, and finally semi-automatically manually annotate the cleaned intelligence data to build a preliminary intelligence corpus; After the whitelist filtering, the intelligence knowledge base set (E, P) is obtained, and m i The dimension vector, m i The current whitelist dimension, E is the vulnerability set, and P is the hacker organization set Step S2, behavior entity modeling Based on the intelligence corpus, threat knowledge extraction and data classification and statistics are performed to realize data clustering analysis and obtain threat intelligence data; a threat knowledge prediction model is built, and the intelligence data is trained using a support vector machine algorithm; When extracting threat knowledge, events, risks, and threat knowledge are extracted in time sequence; a threat knowledge prediction model is built based on event, risk, and threat knowledge triples; In step S2, since the support vector machine algorithm has the problem of blindness in selecting training process parameters in the prediction model, the particle swarm optimization algorithm is used to optimize the selected parameters when building the threat knowledge prediction model, thereby improving the accuracy of the prediction model; The specific steps are as follows: Step S2.1, match the intelligence knowledge base with the built-in feature library, if the matching is successful, terminate the matching process, and store the matching result data; if all the features in the feature library fail to match, terminate the matching; Step S2.2, read the pre-prepared model training sample set, and use the extracted event, risk, and threat knowledge triples as initial training parameters to complete the first training of the support vector machine model, and obtain an initial prediction model; The support vector machine model uses a sub-gradient descent algorithm, and the formula is as follows: Wherein, m is the event set, the current whitelist dimension m i is the i th element in the event set m; x i is the i th risk element, y i is the i th threat knowledge, n is the number of cycles; Step S2.3, continue to read the model training sample set, and use the initial prediction model to complete the situation value prediction of the test sample, and obtain the initial prediction result; Step S2.4, correct the prediction result using the particle swarm algorithm; The fitness evaluation function R of the particle swarm algorithm is as follows: where f 1n is the initial prediction, f 2n is the true state value in the test sample set; The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set; when the fitness evaluation function R meets the entire convergence condition, the initial prediction model is the final prediction model, otherwise the particle swarm algorithm is iterated and the second set of training parameters is transmitted to the support vector machine model, the support vector machine model re-reads the training sample set for the second time to obtain the second generation prediction model, and the test sample set is used to test the precision of the second generation prediction model; in this way, the final prediction model that meets the fitness evaluation function R convergence condition is obtained; and the prediction result is output; Step S3, generate a dynamic threat map Draw threat portraits based on the threat intelligence data obtained after the behavior entity modeling in step S2, and build a threat map.

2. The dynamic threat map generation method based on improved support vector machine according to claim 1, characterized in that: In step S1, a white list mechanism is introduced during the cleaning of the read data; the text data collected and collected is filtered through the white list and then output for use; The public or semi-public text data includes but is not limited to public vulnerability library, public microblog of hacker organization and public microblog data of hacker organization; The whitelist of the public or semi-public text data at least includes the following dimensions: 1) vulnerability intelligence library whitelist dimension: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate and vulnerability submitter vulCommitAuthor; 2) hacker organization whitelist dimension: microblog ID AuthorID, hacker organization ID HackersID, content data published by the hacker organization HackersPublishDesc, ID of the subscribed user SeedID, read count ReadCount, content commented by the hacker organization HackersCommits and user ID focused by the hacker organization HackersFocusUserID.

3. The dynamic threat map generation method based on improved support vector machine according to claim 2, characterized in that: In the step S1, the public vulnerability library includes but is not limited to CVE vulnerability library and SYNK vulnerability library; For the public vulnerability library, a network crawler is written to classify and crawl in the form of XML; the data structure includes six fields of vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate and vulnerability submitter vulCommitAuthor, and a vulnerability set E=(vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor) is obtained; The hacker organization is marked as v, a hacker organization set P is constructed, v∈P; historical data published by the hacker organization v in the past are crawled, and the microblog user ID of each data corresponding to each intelligence like, forward and comment, i.e. the microblog user ID of each data corresponding to each threat intelligence, is taken as subsequent threat graph data.

4. The dynamic threat map generation method based on improved support vector machine according to claim 1, characterized in that: In the step S2, the weighted round robin is adopted by the round robin algorithm to schedule the process of knowledge construction.

5. The dynamic threat map generation method based on improved support vector machine according to claim 1, characterized in that: In the step S3, taking a security event as the ontology, the threat intelligence field ontology structure is created from top to bottom according to the characteristics and rules of the threat intelligence field by abstracting the whole attack process, taking the entity information including but not limited to attack domain name and IP as a tuple, six threat information components of the threat intelligence ontology are proposed, which are security event, threat subject, attack index, observable object, attack method and attack target.

6. An improved support vector machine based dynamic threat graph generation system, characterized in that: The knowledge learning module, The behavior entity modeling module, the dynamic threat graph generation module and the logic storage module; The knowledge learning module reads data from the public or semi-public text data on the Internet as the original corpus, classifies and cleanses the read data, and finally semi-automatically manually marks the cleaned intelligence data to construct a primary intelligence corpus. After the whitelist filtering, the intelligence knowledge base set (E, P) is obtained, and m i The vector, m i E is the current whitelist dimension, E is the vulnerability set, and P is the hacker organization set The behavior entity modeling module comprises a threat knowledge extraction submodule, a threat knowledge construction submodule and a task scheduling submodule; The threat knowledge extraction submodule extracts threat knowledge from intelligence corpus and classifies and counts data, realizes data clustering analysis and obtains threat intelligence data; The threat knowledge construction submodule is responsible for constructing a threat knowledge prediction model and training intelligence data by using a support vector machine algorithm; The task scheduling submodule schedules the process of knowledge construction by using a weighted round robin method through a round algorithm; When the threat knowledge extraction submodule extracts threat knowledge, events, risks and threat knowledge are extracted in sequence in the time dimension; a threat knowledge prediction model is constructed based on the event, risk and threat knowledge triplets; The threat knowledge construction submodule optimizes the selected parameters by using a particle swarm optimization algorithm, thereby improving the accuracy of the prediction model; the specific steps are as follows: Step S2.1, the intelligence knowledge base is matched with the built-in feature library, if the matching is successful, the matching process is terminated, and the matching result data is marked and stored; if all the features in the feature library are matched, the matching is terminated; Step S2.2, read the pre-prepared model training sample set, and use the extracted event, risk and threat knowledge triplets as the initial training parameters to complete the first training of the support vector machine model, and obtain the initial prediction model; The support vector machine model uses a sub-gradient descent algorithm, and the formula is as follows: Wherein, m is the event set, the current whitelist dimension m i is the i-th element in the event set m; x i is the i-th risk element, y i is the i-th threat knowledge, n is the number of cycles; Step S2.3, continue to read the model training sample set, and use the initial prediction model to complete the situation value prediction of the test sample, and obtain the initial prediction result; Step S2.4, correct the prediction result by using the particle swarm algorithm; The fitness evaluation function R of the particle swarm algorithm is as follows: where f 1n is the initial prediction, f 2n is the true state value in the test sample set; The fitness evaluation function R is used to calculate the error between the initial prediction result and the true situation value in the test sample set; when the fitness evaluation function R meets the entire convergence condition, the initial prediction model is the final prediction model, otherwise the particle swarm algorithm is iterated and the second group of training parameters is transmitted to the support vector machine model, the support vector machine model re-reads the training sample set for the second time to obtain the second generation prediction model, and the test sample set is used to test the accuracy of the second generation prediction model; in this way, the final prediction model meeting the fitness evaluation function R convergence condition is obtained; output the prediction result; The dynamic threat graph generation module performs threat characterization on the threat intelligence data obtained after the behavior entity modeling, and constructs a threat graph; The logical storage module comprises two types of storage modes, namely a distributed MySQL database cluster and a distributed HBASE database; the distributed MySQL database cluster is used to save the sample set of model training and the determination result of the attack; the distributed HBASE database is used to save the signal capture amount related information.

7. The improved support vector machine based dynamic threat map generation system as claimed in claim 6, wherein: The knowledge learning module is provided with a whitelist mechanism, and the text data collected and collected is filtered by the whitelist and then output for use. The public or semi-public text data includes but is not limited to public vulnerability library, public microblog of hacker organization and public microblog data of hacker organization; The whitelist of public or semi-public text data at least includes the following dimensions: 1) vulnerability intelligence library whitelist dimension: vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate and vulnerability submitter vulCommitAuthor; 2) hacker organization whitelist dimension: microblog ID AuthorID, hacker organization ID HackersID, content data published by hacker organization HackersPublishDesc, ID of subscribed user SeedID, read count ReadCount, content commented by hacker organization HackersCommits and user ID focused by hacker organization HackersFocusUserID; The public vulnerability library includes but is not limited to CVE vulnerability library and SYNK vulnerability library; For the public vulnerability library, a network crawler is written to classify and crawl in the form of XML; the data structure includes six fields of vulnerability number vulNumber, vulnerability content vulDesc, vulnerability level vulLevel, vulnerability disclosure time vulCommitDate, vulnerability update time vulUpdate and vulnerability submitter vulCommitAuthor, and a vulnerability set E=(vulNumber, vulDesc, vulLevel, vulCommitDate, vulUpdate, vulCommitAuthor) is obtained; The hacker organization is marked as v, a hacker organization set P is constructed, v∈P; historical data published by the hacker organization v in the past are crawled, and the microblog user ID of each data corresponding to each intelligence like, forward and comment, i.e. the microblog user ID of clicking corresponding threat intelligence, is taken as subsequent threat graph data.

8. The improved support vector machine based dynamic threat graph generation system as claimed in claim 6, wherein: The dynamic threat graph generation module takes a security event as the ontology, abstracts the entire attack process, takes entity information including but not limited to attack domain name and IP as a tuple, creates a threat intelligence field ontology structure from top to bottom according to the characteristics and rules of the threat intelligence field, proposes six threat information components for the threat intelligence ontology, and the six threat information components are respectively a security event, a threat subject, an attack index, an observable object, an attack method and an attack target.

Images