Incremental extraction and analysis method for large-scale heterogeneous vulnerability data

Abstract

The application discloses a large-scale heterogeneous-oriented incremental extraction and analysis method for vulnerability data, realizes automatic extraction and real-time updating of vulnerability data information based on a heterogeneous crawler framework, and analyzes vulnerability distribution, evolution track and development trend.The method comprises the steps of incremental extraction of vulnerability data, preprocessing of vulnerability data, standardization of vulnerability data, storage of vulnerability data and analysis of vulnerability data, extracts vulnerability data information from different sources, and comprises domestic and foreign public vulnerability databases such as NVD, CNNVD, the national information security vulnerability sharing platform CNVD, the commercial vulnerability database Snyk, the open source vulnerability report platform GoogleOSV, and greatly expands the range of vulnerability data sources compared with existing work, solves the problems of long data updating period of single data source, incomplete coverage of vulnerability data range, missing of vulnerability information items and the like, and improves the data quality of vulnerability information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, in particular to a large-scale heterogeneous vulnerability data incremental extraction and analysis method. BACKGROUND

[0002] In the current Internet era, vulnerability is an important aspect of information security field, and the research on vulnerability is a popular research topic in the field of information security. The basis of vulnerability research work is large-scale extraction and analysis of vulnerability data. With the rapid development of the Internet and the continuous upgrading of network attack means, the diversification of vulnerability information sources, the explosive growth of vulnerability numbers and the accelerated iteration of vulnerability reports have put forward higher requirements for vulnerability information extraction and analysis.

[0003] There are many existing vulnerability databases, and the vulnerability data formats and vulnerability report publishing times of different databases are different, which are difficult to integrate and use. At present, there is no perfect and feasible method to extract and integrate large-scale vulnerability data from different database sources and realize incremental updating. At the same time, there is a lack of systematic analysis of large-scale vulnerability data distribution and development trend. Therefore, it is of great significance to study the large-scale heterogeneous vulnerability data extraction and analysis method for improving the extraction efficiency and analysis ability of vulnerability data. SUMMARY

[0004] The purpose of the present application is to solve the problems of large differences in data formats between existing heterogeneous vulnerability databases, low vulnerability data extraction efficiency and difficulty in systematic vulnerability data analysis, and to propose a large-scale heterogeneous incremental vulnerability data extraction and analysis method.

[0005] The present application is realized by the following technical solutions:

[0006] The present application discloses a large-scale heterogeneous vulnerability data incremental extraction and analysis method, characterized by realizing automatic extraction and real-time updating of vulnerability data information based on a heterogeneous crawler framework, and analyzing vulnerability distribution, evolution track and development trend. The method comprises the following steps:

[0007] 1) Incremental extraction of vulnerability data; incremental extraction of vulnerability data from different sources, wherein the vulnerability data includes structured vulnerability data such as CVE (Common Vulnerabilities and Exposures) number and unstructured vulnerability data such as affected version information;

[0008] 2) Vulnerability data preprocessing; based on the vulnerability data extracted in step 1), the structured vulnerability data and unstructured vulnerability data are preprocessed;

[0009] 3) Vulnerability data standardization; based on the vulnerability data after preprocessing in step 2), the vulnerability data of different sources is processed in a standardized format;

[0010] 4) Vulnerability data storage; based on steps 1), 2) and 3), the vulnerability data after vulnerability data preprocessing and vulnerability data standardization is stored in a local database;

[0011] 5) Vulnerability data analysis; based on the vulnerability data stored in step 4), the distribution, evolution track and development trend of the vulnerability are analyzed.

[0012] As a further improvement, the step 1) of the present application comprises the following steps:

[0013] (1) determining the target website for extracting vulnerability data;

[0014] (2) writing corresponding crawler scripts according to the data structure of different websites;

[0015] (3) using the crawler to access the vulnerability information page in the target website, and parsing to obtain the detail link of the whole website data;

[0016] (4) generating the md5 data fingerprint of the corresponding website vulnerability information page, and storing it in the local database;

[0017] (5) comparing the md5 data fingerprints of the latest two vulnerability information pages in the local database, if they are not the same, then extracting the vulnerability data;

[0018] (6) periodically repeating steps (1) to (5) to realize incremental extraction of large-scale heterogeneous vulnerability data information.

[0019] As a further improvement, the vulnerability data in step 1) of the present application includes: vulnerability name, CVE number, CWE number, vulnerability level, vulnerability score, vulnerability submission time, vulnerability disclosure time, vulnerability repair time, vulnerability introduction, reference information, CVSS vector, CVSS score, affected software package name, affected version range information, affected software package type.

[0020] As a further improvement, the vulnerability data preprocessing operation in step 2) of the present application includes screening out vulnerability data with missing information items, and removing non-ASCII characters and line breaks in the unstructured vulnerability data.

[0021] As a further improvement, the vulnerability data standardization operation in step 3) of the application includes merging and deduplicating the heterogeneous vulnerability data in step 1) with CVE number as the unique feature, and standardizing the heterogeneous source data into a standard vulnerability data format and content.

[0022] As a further improvement, the standard vulnerability data format and content of the application includes: vulnerability name, CVE number, CWE number, vulnerability level, vulnerability score, vulnerability disclosure time, vulnerability introduction, reference information, affected software package name, affected version range information, and affected software package type.

[0023] As a further improvement, the vulnerability data storage operation in step 4) of the application includes:

[0024] (1) Establishing a local vulnerability database and establishing corresponding table entries according to the standard vulnerability data format;

[0025] (2) Using the pipeline operation in the crawler framework to persistently store the heterogeneous vulnerability data after vulnerability data preprocessing and vulnerability data standardization.

[0026] As a further improvement, the vulnerability data analysis operation in step 5) of the application includes:

[0027] (1) Analyzing the distribution of vulnerability data in different programming language package managers;

[0028] (2) Analyzing the change of vulnerability number in mainstream programming language package managers over time;

[0029] (3) Analyzing the development trend of vulnerabilities in different ecosystems.

[0030] As a further improvement, the target website of the application includes: NVD (National Vulnerability Database, US National Vulnerability Database), CNVD (China National Vulnerability Database, National Information Security Vulnerability Sharing Platform), CNNVD (China National Vulnerability Database of Information Security, National Information Security Vulnerability Database), Snyk, Google OSV.

[0031] The beneficial effects of the application are as follows:

[0032] (1) The heterogeneity of the vulnerability data source: the present application extracts vulnerability data information of different sources, including domestic and foreign public vulnerability databases such as NVD, CNNVD, national information security vulnerability sharing platform CNVD, commercial vulnerability database Snyk, open source vulnerability reporting platform GoogleOSV, and compared with the existing work, the range of vulnerability data source is greatly expanded, the problems of long data update cycle, incomplete coverage of vulnerability data range, and missing of vulnerability information items in single data source are solved, and the data quality of vulnerability information is improved.

[0033] (2) The standardized processing of vulnerability data: the present application screens out the accurate vulnerability data of information items and removes the non-structured vulnerability data of non-ASCII characters and line breaks through the vulnerability data preprocessing operation; through the vulnerability data standardization operation, the important information of heterogeneous vulnerability data is extracted and unified into a standardized format. Through the standardized processing of vulnerability data, the heterogeneous vulnerability data is converted into a unified data format which can be directly used.

[0034] (3) The persistent storage and incremental update of vulnerability data: the present application realizes the persistent storage of vulnerability data by importing the vulnerability data of the target website into the local database through the pipelines operation. The present application generates the md5 data fingerprint of the vulnerability information page of the corresponding website, compares it with the data fingerprint of the last extraction, and judges whether the website vulnerability information is updated, so as to realize the incremental update of vulnerability data.

[0035] (4) The systematic analysis of vulnerability data: based on the large-scale heterogeneous vulnerability data extracted by the present application, the distribution of vulnerability data in different programming language package managers, the change of vulnerability quantity in mainstream programming language package managers with time, and the development trend of vulnerability in different ecosystems are analyzed, which makes up for the defects of insufficient vulnerability data and inaccurate analysis results in related analysis work. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of an embodiment of the vulnerability data incremental extraction and analysis method for large-scale heterogeneous vulnerability data in the present application;

[0037] Figure 2 is a flowchart of an embodiment of the vulnerability data incremental extraction in the present application; DETAILED DESCRIPTION

[0038] The specific technical solutions of the present application will be further described below in combination with the drawings of the specification:

[0039] Figure 1 is a flowchart of an embodiment of the vulnerability data incremental extraction and analysis method for large-scale heterogeneous vulnerability data, which specifically includes the following steps:

[0040] (1) Incrementally extracting vulnerability data of different sources, including structured vulnerability data numbers and unstructured vulnerability data;

[0041] Specifically, in one embodiment, as shown in Figure 2 the step (1) includes the following steps:

[0042] 1) determining a target website for extracting vulnerability data;

[0043] For example, the target website includes NVD, CNVD, CNNVD, Snyk, GoogleOSV, etc. Compared with the CVE official website, the present embodiment collects data from public and commercial vulnerability database platforms, and obtains more detailed vulnerability data. Compared with the prior art, the present embodiment greatly expands the range of vulnerability data sources, solves the problems of long data update cycle of single data source, incomplete coverage of vulnerability data range, and missing vulnerability information items, and improves the data quality of vulnerability information.

[0044] 2) writing corresponding crawler scripts according to the data structures of different websites;

[0045] Generally, the data structures of different vulnerability database websites are different, and corresponding extraction scripts need to be written for different websites under the same crawler framework to collect vulnerability data.

[0046] In one embodiment, since the server of different vulnerability database websites is located in different regions, there may be a situation of too long response time or even access error, and a proxy needs to be set to perform asynchronous and concurrent extraction on the target website. Specifically, the maximum access time limit and the maximum access number limit can be set to terminate abnormal access.

[0047] In one embodiment, if the target website sets an anti-crawling mechanism, IP proxy pool technology can be used to change different IPs for crawling to prevent IP restriction problems caused by a large number of requests in a short time.

[0048] 3) using a crawler to access the vulnerability information page in the target website to parse the detail link of the full-site data;

[0049] In one embodiment, by accessing the vulnerability information page in the target website, a list of detail links of full-site data can be extracted, and the obtained response is processed by the spider module of the crawler to analyze and extract data, and the item field is transmitted to the itempipeline for processing.

[0050] 4) generating md5 data fingerprints of the vulnerability information pages of the corresponding website and storing them in a local database;

[0051] In one embodiment, the md5 data fingerprint is generated by a hash function, and the md5 data fingerprint of the corresponding website vulnerability information page is obtained at each extraction of vulnerability information and stored in the local database, which is used for subsequent steps to determine whether the vulnerability data of the website is updated.

[0052] 5) Compare the md5 data fingerprints of the latest two vulnerability information pages in the local database. If they are not the same, perform vulnerability data extraction;

[0053] Generally, due to the characteristics of md5 data fingerprint, it is almost impossible for different pages to have the same md5 data fingerprint, so any new vulnerability data in the vulnerability information page will cause the md5 data fingerprint to change. Therefore, using the md5 data fingerprint to determine whether the vulnerability data is updated has a theoretical basis.

[0054] In one embodiment, after generating the md5 data fingerprint of the website vulnerability information page and storing it in the local database for the first time, since there is only one md5 data fingerprint at this time, vulnerability data extraction is directly performed. In addition, if the comparison results of the md5 data fingerprints of the latest two vulnerability information pages in the local database are the same, it means that the vulnerability data of the corresponding website has not been updated, and vulnerability data extraction is not performed; otherwise, vulnerability data extraction is performed.

[0055] 6) Repeat steps 1) to 5) periodically to achieve incremental extraction of large-scale heterogeneous vulnerability data information.

[0056] In one embodiment, the steps 1) to 5) are periodically executed, the vulnerability information page of the corresponding website is accessed, and it is determined whether to perform vulnerability data extraction by judging whether the vulnerability data is updated, thereby achieving incremental extraction of large-scale heterogeneous vulnerability data information.

[0057] (2) Preprocess structured vulnerability data and unstructured vulnerability data;

[0058] Specifically, in one embodiment, the vulnerability data extracted through step (1) may lack key vulnerability information items, and the structured vulnerability data and unstructured vulnerability data missing vulnerability information items are screened out through the vulnerability data preprocessing operation.

[0059] Specifically, in one embodiment, the unstructured vulnerability data extracted through step (1) includes some non-ASCII characters and line breaks, and in order to facilitate subsequent persistent storage and vulnerability data analysis, the non-ASCII characters and line breaks in the unstructured vulnerability data are deleted.

[0060] (3) Standardize the format of vulnerability data from different sources;

[0061] Generally, since the same CVE will be included by different vulnerability database platforms, the results of heterogeneous vulnerability data information extraction will have duplication, and the vulnerability data standardized format processing combines and removes the duplicate processing of vulnerability data of different vulnerability database platforms by taking the CVE number as the unique feature.

[0062] Specifically, in one embodiment, for the same CVE but different source vulnerability data information, the vulnerability data information items are compared in turn, and the union of the contents of the vulnerability data information items is taken as the result of combining and removing duplicates.

[0063] Specifically, in one embodiment, the content format of the vulnerability data information items given by different vulnerability database platforms may be different, such as the affected version range information, which is given in the form of an inequality on Snyk, in the form of CPE (Common Platform Enumeration) on NVD, and in the form of enumeration on GoogleOSV. The vulnerability data standardized format processing unifies the presentation formats of different vulnerability data information items.

[0064] (4) persistently store the vulnerability data that has been preprocessed and standardized into a local database;

[0065] Specifically, in one embodiment, a local MySQL vulnerability database is first established, and corresponding table items are established according to the standard vulnerability data format; the pipelines operation in the crawler framework is used to open the MySQLPipeline file, involving three methods of open_spider, process_item, and close_spider. The open_spider is used to create a new file or connect to the database and specify the encoding method when the spider starts running; the process_item method is used for data processing work, which returns a dict, item with data; the close_spider is used to close the persistent storage link when the program runs.

[0066] (5) based on the stored vulnerability data, analyze the distribution, evolution track, and development trend of the vulnerability.

[0067] Specifically, in one embodiment, the step (5) includes the following steps:

[0068] 1) analyze the distribution of vulnerability data in different package managers of programming languages;

[0069] Specifically, in one embodiment, the contents of the vulnerability information data items such as "affected software package type", "CVE number", "vulnerability level", "vulnerability introduction" and the like in the local vulnerability database are extracted, so as to obtain the distribution of the number of vulnerabilities in different programming language package managers and the distribution of vulnerability types in each package manager, and thus the security situation of the current package managers of different programming languages and the main reasons for the introduction of vulnerabilities by the package managers can be obtained.

[0070] Specifically, in one embodiment, the number of vulnerabilities in Maven (Java package manager) is 5622, the number of vulnerabilities in Npm (Node.js package manager) is 3411, and the number of vulnerabilities in Pypi (Python package manager) is 2810.

[0071] 2) Analyzing the change of the number of vulnerabilities in the package managers of mainstream programming languages over time;

[0072] Specifically, in one embodiment, Maven, Npm, Pypi, Nuget, Packagist, Godoc, Swift, and Rubygems, eight mainstream programming language package managers, are selected in the "affected software package type" of the local vulnerability database, and the corresponding "CVE number" and "vulnerability disclosure time" are extracted, and the number of disclosed vulnerabilities in each programming language package manager in the past ten years is counted in turn, and thus the evolution trajectory of vulnerabilities in the mainstream programming language package managers can be obtained.

[0073] 3) Analyzing the development trend of vulnerabilities in different ecosystems.

[0074] Specifically, in one embodiment, on the basis of the steps 1) and 2), and in combination with the distribution of vulnerability data in different programming language package managers and the change of the number of vulnerabilities in the mainstream programming language package managers over time, the security situation and the historical evolution of vulnerabilities in different ecosystems can be obtained, and thus the development trend of vulnerabilities in different ecosystems can be analyzed.

[0075] Those skilled in the art can understand that the above description is only a single example of the application and is not intended to limit the application, although the application has been described in detail with reference to the foregoing examples, and those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent replacements for part of the technical features. Any modification, equivalent replacement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A method for incremental extraction and analysis of vulnerability data in large-scale heterogeneous environments, characterized in that, Based on a heterogeneous web crawler framework, this system automates the extraction and real-time updating of vulnerability data, and analyzes the distribution, evolution, and development trends of vulnerabilities. The process includes the following steps: 1) Incremental extraction of vulnerability data; 1.1) Identify the target websites for extracting vulnerability data. The target websites include domestic and international publicly available vulnerability databases such as NVD, CNNVD, CNVD, commercial vulnerability database Snyk, and open-source vulnerability reporting platform Google OSV. NVD stands for National Vulnerability Database, the U.S. national vulnerability database; CNVD stands for China National Vulnerability Database, the national information security vulnerability sharing platform; CNNVD stands for China National Vulnerability Database of Information Security, the national information security vulnerability database. 1.2) Write corresponding crawler scripts based on the data structure of different websites; 1.3) Use a web crawler to access the vulnerability information page on the target website and parse it to obtain the detailed links of the entire site's data; 1.4) Generate the MD5 data fingerprint of the corresponding website vulnerability information page and store it in the local database; 1.5) Compare the MD5 data fingerprints of the two most recent vulnerability information pages in the local database. If they are different, extract the vulnerability data. The vulnerability data includes structured vulnerability data and unstructured vulnerability data. The structured vulnerability data includes the CVE number. 1.6) Periodically repeat steps 1.1) to 1.5) to achieve incremental extraction of large-scale heterogeneous vulnerability data information; 2) Vulnerability data preprocessing; Based on the vulnerability data extracted in step 1), filter out vulnerability data with missing information items, and remove non-ASCII characters and newline characters from unstructured vulnerability data; 3) Vulnerability Data Standardization; Based on the preprocessed vulnerability data in step 2), the CVE number is used as the unique feature for merging and deduplication, and unstructured vulnerability data from different data sources are uniformly converted into a standard format to form standard vulnerability data; The standard vulnerability data format and content include: vulnerability name, CVE number, CWE number, vulnerability level, vulnerability score, vulnerability disclosure time, vulnerability description, reference information, affected software package name, affected version range information, and affected software package type. The unstructured vulnerability data includes Snyk's inequality format, NVD's CPE format, and Google OSV's enumeration format of affected version range information; 4) Vulnerability data storage; Establish a local vulnerability database, create corresponding table entries according to the standard vulnerability data format in step 3), and use the pipeline operation in the crawler framework to persistently store the preprocessed and standardized vulnerability data. 5) Vulnerability data analysis; Based on the vulnerability data stored in step 4), the following multi-dimensional analysis is performed: 5.1) Analyze the distribution of vulnerability data in package managers of different programming languages, including Maven, Npm, Pypi, Nuget, Packagist, Godoc, Swift, and Rubygems; 5.2) Analyze the changes in the number of vulnerabilities in package managers of mainstream programming languages ​​over time.

2. The incremental extraction and analysis method for large-scale heterogeneous vulnerability data according to claim 1, characterized in that, The vulnerability data standardization operation in step 3) includes merging and deduplicating the heterogeneous vulnerability data in step 1) using the CVE number as the unique feature, and standardizing the heterogeneous source data into a standard vulnerability data format and content.

3. The incremental extraction and analysis method for large-scale heterogeneous vulnerability data according to claim 1, characterized in that, The vulnerability data storage operation in step 4) includes: 1) Establish a local vulnerability database and create corresponding table entries according to the standard vulnerability data format; 2) Utilize pipeline operations in the crawler framework to persistently store heterogeneous vulnerability data after vulnerability data preprocessing and standardization.

Images