Multi-source unstructured threat intelligence collection and fusion method based on key feature comparison

Abstract

A multi-source unstructured threat intelligence collection and fusion method based on key feature comparison, which method relates to the technical field of information security. The method comprises: connecting to a plurality of data sources of threat intelligence by means of APIs (S1); acquiring multi-source unstructured threat intelligence from the different data sources (S2); performing normalization processing on the multi-source unstructured threat intelligence by means of a natural language processing technology, and converting multi-source unstructured threat intelligence data into structured data (S3); and acquiring key features of all structured threat intelligence, calculating the similarity between the key features and key features of a preset standard, and if the similarity meets a requirement, fusing all the structured threat intelligence to form a threat intelligence cluster, and sending the fused threat intelligence cluster to a corresponding user (S4). The method improves the collection efficiency and saves time, and can fuse all the multi-source unstructured threat intelligence, thereby improving the query coverage rate and ensuring the integrity and accuracy of data.

Description

A multi-source unstructured threat intelligence collection and fusion method based on key feature comparison Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a multi-source unstructured threat intelligence collection and fusion method based on key feature comparison. Background Art

[0002] Threat intelligence is a type of knowledge information that describes threats based on evidence, including contextual information related to the threat, the methods and mechanisms used by the threat, threat-related indicators, attack impacts, and response action recommendations.

[0003] Threat intelligence collection is the process of gathering dispersed threat intelligence and extracting the required information from it. Typically, intelligence collectors or analysts must search multiple search engines, platforms, and websites to gather relevant information. Furthermore, they must use or change search keywords as needed, and manually filter and organize the results to generate a threat intelligence report. This entire process is extremely time-consuming and labor-intensive.

[0004] Furthermore, in actual applications, due to the limited coverage of threat intelligence data provided by various threat intelligence data providers or data sources, clients often rely on multiple sources of threat intelligence data to achieve effective network security protection. However, due to differences in the production methods and data processing capabilities of threat intelligence data from different sources, the attributes or descriptions of the threat intelligence data may vary. Even the same intelligence value generated by different sources may produce inconsistent results within the threat intelligence data. These results are the basis for determining how to use the threat intelligence data. If the same intelligence value has multiple different results, it becomes impossible to determine how to use the threat intelligence data.

[0005] Summary of the Invention

[0006] The purpose of this invention is to provide a multi-source unstructured threat intelligence collection and fusion method based on key feature comparison, which can solve the problems existing in the background technology. The specific technical solution is as follows:

[0007] A multi-source unstructured threat intelligence collection and fusion method based on key feature comparison includes:

[0008] Connect to multiple threat intelligence data sources through API interfaces;

[0009] Obtain multi-source unstructured threat intelligence from different data sources;

[0010] Normalize the multi-source unstructured threat intelligence through natural language processing technology, and convert the multi-source unstructured threat intelligence data into structured data;

[0011] The key features of all the structured threat intelligence are obtained and the similarity is calculated with the key features of the preset standard. If the similarity meets the requirements, all the structured threat intelligence are fused to form a threat intelligence cluster, and the fused threat intelligence cluster is sent to the corresponding user.

[0012] Preferably, the data sources connected to multiple threat intelligence via an API interface are multiple search engines, social networks, paper websites, and online forum websites.

[0013] Preferably, multi-source unstructured threat intelligence is obtained from different data sources, including:

[0014] When acquiring multi-source unstructured threat intelligence, determine whether the URL and IP of the data source are safe, cut off the URLs and IPs that do not meet the standards, stop acquiring the multi-source unstructured threat intelligence, and complete the preliminary screening.

[0015] Preferably, the normalizing of the multi-source unstructured threat intelligence using natural language processing technology to convert the multi-source unstructured threat intelligence data into structured data includes:

[0016] By representing unstructured data intelligence data through existing standards, structured threat intelligence data is formed.

[0017] Preferably, the key features include at least key fields and intelligence topics.

[0018] Preferably, the similarity calculation is implemented through a bag-of-words model, a word vector model, or a BERT model in natural language processing technology.

[0019] Preferably, the key feature further includes categories, and the structured threat intelligence can be integrated through the categories of threat intelligence.

[0020] Preferably, the key features of the preset standards include update time, source, discovery time, type, key fields, completeness, difference, intelligence data content and intelligence subject.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. In the present invention, unstructured threat intelligence is extracted from multiple data sources through the API interface, and unstructured threat intelligence data can be collected from multiple data sources at the same time, ensuring that the collected data has a wide coverage and complete data, while avoiding manual collection, improving collection efficiency and saving time.

[0023] 2. In the present invention, natural language processing technology can be used to process multi-source unstructured threat intelligence according to existing specifications and transform it into structured data. It can also fuse the scattered multi-source unstructured data to form a threat intelligence cluster, which is helpful for the fusion of multi-source unstructured threat intelligence.

[0024] 3. In the present invention, after converting unstructured threat intelligence into structured threat intelligence, the collected threat intelligence data is compared with the key features of preset standard threat intelligence data for similarity, and all threat data containing the same key fields and topics are fused and stored into a threat intelligence data cluster, and sent to the user. The user can browse the threat intelligence data of related topics, and the data is complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0026] FIG1 is a system principle diagram of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] Referring to FIG1 , the present invention discloses a multi-source unstructured threat intelligence collection and fusion method based on key feature comparison, including:

[0032] Connect to multiple threat intelligence data sources through API interfaces;

[0033] Specifically, the above-mentioned API is an extensible interface that can be connected to multiple data sources, ensuring that a relatively complete set of unstructured threat intelligence data can be collected, where the data sources include various search engines, social networks, paper websites, online forum websites, or other platforms or websites that include threat intelligence.

[0034] Obtain multi-source unstructured threat intelligence from different data sources;

[0035] In the above steps, when acquiring multi-source unstructured threat intelligence, links may be directed to prohibited or illegal websites, posing the risk of illegal or erroneous threat intelligence. Therefore, when connecting to the data source, the URL and IP address of the threat intelligence provided by the data source are determined to be safe. URLs and IP addresses that do not meet the standards are disconnected, and the acquisition of multi-source unstructured threat intelligence is stopped, completing the initial screening of unstructured threat intelligence.

[0036] The multi-source unstructured threat intelligence is standardized through natural language processing technology, and the multi-source unstructured threat intelligence data is converted into structured data; when searching for unstructured threat intelligence, an input text can be set, keywords can be entered, or voice input can be used. Natural language processing technology will process these texts and voices to generate relevant semantics or texts, and then search in the data source, without the need for professional personnel to write professional terms before collecting data.

[0037] The key features of all the structured threat intelligence are obtained and the similarity is calculated with the key features of the preset standard. If the similarity meets the requirements, all the structured threat intelligence are fused to form a threat intelligence cluster, and the fused threat intelligence cluster is sent to the corresponding user.

[0038] As a preferred solution, the multi-source unstructured threat intelligence is normalized by natural language processing technology to convert the multi-source unstructured threat intelligence data into structured data, including:

[0039] By representing unstructured data intelligence data through existing standards, structured threat intelligence data is formed.

[0040] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely related to the study of linguistics, yet it also differs significantly. Natural language processing is not simply the study of natural language in general, but rather the development of computer systems, particularly software systems, that can effectively implement natural language communication. Therefore, it is a branch of computer science.

[0041] It should be noted that the key features include at least key fields and intelligence topics. If the key fields and intelligence topics are similar, then the threat intelligence information belonging to the same content can be merged.

[0042] Specifically, the similarity calculation is implemented through a bag-of-words model, a word vector model, or a BERT model in natural language processing technology. The above models are all existing mature calculation models and will not be described in detail here.

[0043] In this embodiment, the key feature also includes categories, and the structured threat intelligence can be integrated according to the categories of threat intelligence. Specifically, the categories of threat intelligence include IP threat intelligence, virus hash threat intelligence, malicious domain threat intelligence, and malicious site threat intelligence.

[0044] In addition, the key features of the preset standards include update time, source, discovery time, type, key fields, completeness, difference, intelligence data content and intelligence subject.

[0045] To sum up, the present invention can collect unstructured threat intelligence data from multiple data sources at the same time, ensuring that the collected data has a wide coverage and complete data, avoiding manual collection, improving collection efficiency, and saving time; in addition, through natural language processing technology, multi-source unstructured threat intelligence can be processed and converted into structured data according to existing specifications, and scattered multi-source unstructured data can be integrated to form a threat intelligence cluster, which is helpful for the integration of multi-source unstructured threat intelligence; at the same time, after converting unstructured threat intelligence into structured threat intelligence, the collected threat intelligence data is compared with the preset standard threat intelligence data key features for similarity, and all threat data containing the same key fields and themes are integrated and stored into a threat intelligence data cluster, and sent to users. Users can browse the threat intelligence data of related topics, and the data is complete and intact.

[0046] Those skilled in the art will appreciate that the units of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0047] In the embodiments provided by the present invention, it should be understood that the division of units is merely a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0048] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0049] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program code.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A multi-source unstructured threat intelligence collection and fusion method based on key feature comparison, characterized in that Including: Connecting to multiple threat intelligence data sources through an API interface; Obtaining multi-source unstructured threat intelligence from different data sources; Normalizing the multi-source unstructured threat intelligence through natural language processing technology, and converting the multi-source unstructured threat intelligence data into structured data; Calculating the similarity between the key features of all the structured threat intelligence and the key features of a preset standard. If the similarity meets the requirements, all the structured threat intelligence will be fused to form a threat intelligence cluster, and the fused threat intelligence cluster will be sent to the corresponding users.

2. The multi-source unstructured threat intelligence collection and fusion method based on key feature comparison according to claim 1, wherein Among the data sources connected to multiple threat intelligence through the API interface, the data sources are various search engines, social networks, paper websites, and online forum websites.

3. The multi-source unstructured threat intelligence collection and fusion method based on critical feature comparison according to claim 1, wherein, Obtaining multi-source unstructured threat intelligence from different data sources, including: When obtaining multi-source unstructured threat intelligence, judging whether the URLs and IPs of the data sources are secure, cutting off the non-compliant URLs and IPs, stopping the acquisition of the multi-source unstructured threat intelligence, and completing the preliminary screening.

4. The multi-source unstructured threat intelligence collection and fusion method based on key feature comparison according to claim 1, characterized in that, Normalizing the multi-source unstructured threat intelligence through natural language processing technology and converting the multi-source unstructured threat intelligence data into structured data, including: Forming structured threat intelligence data by representing the unstructured data intelligence data through existing specifications.

5. The multi-source unstructured threat intelligence collection and fusion method based on key feature comparison according to claim 1, characterized in that The key features at least include keyword fields and intelligence topics.

6. The multi-source unstructured threat intelligence collection and fusion method based on critical feature comparison according to claim 1, wherein The similarity calculation is implemented through the bag-of-words model or the word vector model or the BERT model in natural language processing technology.

7. The multi-source unstructured threat intelligence collection and fusion method based on critical feature comparison according to claim 1, wherein, The key features also include categories, and the structured threat intelligence can also be fused by the categories of the threat intelligence.

8. The multi-source unstructured threat intelligence collection and fusion method based on critical feature comparison according to claim 1, wherein, The key features of the preset standard include update time, source, discovery time, type, keyword fields, integrity, difference degree, intelligence data content, and intelligence topic.

Images