Knowledge graph-based threat intelligence analysis method and system

Abstract

The application provides a threat intelligence analysis method and system based on a knowledge graph. Since the embedding layer processing of entity extraction is improved on the basis of existing threat intelligence analysis, the entity category and boundary can be accurately labeled, and the hidden state is obtained by improving the LSTM layer. Then, the mapping relationship of the related entity is predicted by introducing the label transition probability algorithm, so that the deficiency that the existing knowledge graph cannot obtain implicit knowledge when performing semantic retrieval is overcome, and semantic retrieval and reasoning analysis based on the knowledge graph are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a threat intelligence analysis method and system based on a knowledge graph. BACKGROUND

[0002] Threat intelligence analysis is based on network security big data, and stands in the perspective of the attacker to summarize and analyze potential threat information that the defense party will face, in order to improve the network protection capability of the defense party. However, the existing technology cannot accurately label entity categories and boundaries, and the existing knowledge graph cannot obtain implicit knowledge when performing semantic retrieval.

[0003] Therefore, there is an urgent need for a targeted threat intelligence analysis method and system based on a knowledge graph. SUMMARY

[0004] The purpose of the present application is to provide a threat intelligence analysis method and system based on a knowledge graph, which improves the embedding layer processing of entity extraction on the basis of existing threat intelligence analysis, can accurately label entity categories and boundaries, and improves the hidden state obtained by the LSTM layer, and then predicts the mapping relationship of the related entities by introducing the label transition probability algorithm.

[0005] In a first aspect, the present application provides a threat intelligence analysis method based on a knowledge graph, which comprises:

[0006] Collecting state information, domain name information, link addresses and message data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data;

[0007] Performing initialization processing on the open source threat intelligence data, using support vector machine algorithm for text classification based on theme, keyword and length as features, filtering out noise data in the open source threat intelligence data, and performing segmentation processing in units of sentences, automatically labeling identification information, and obtaining a threat intelligence library;

[0008] Extracting sentences from the threat intelligence library in a predetermined order and inputting them into an entity extraction module;

[0009] The entity extraction module inputs the affiliated first word, entity subsequent word and non-entity word into different embedding layers according to the label, and inputs them into the corresponding LSTM layer after processing, the processing including reducing the dimension of the input sentence and adding an index to the first word, the index pointing to a word indicating the end of the entity, the pointing being based on a loss function of the dimension reduction detection, and the word indicating the end of the entity being obtained by solving the optimal solution of the loss function.

[0010] After the vectors are exchanged with each other through the adjacent LSTM layers, a current hidden vector is calculated, the hidden vector is divided into a forward hidden vector and a backward hidden vector, the forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and the hidden state is input into a decoding layer, the decoding layer introduces a transition probability algorithm of a label, takes the label as a label item, and predicts a mapping relationship of the related entity according to a calculated probability value;

[0011] The mapping relationship of the related entity is input into a visualization module to display a knowledge graph of threat intelligence entities, Cypher statements are used to query stored data, and the knowledge graph is provided to a user for decision-making.

[0012] With reference to the first aspect, in a first possible implementation manner of the first aspect, the collecting includes giving different scores to different information sources according to historical records of the information sources; and further includes focusing on collecting information corresponding to a pre-set intelligence type, and dynamically setting information with low relevance to the intelligence type as redundant information and removing the redundant information in the initialization processing.

[0013] With reference to the first aspect, in a second possible implementation manner of the first aspect, the collecting includes extracting elements, judging whether the discovered elements are related to a current hot security event, marking a hot security event summary in the elements if the elements are related to the hot security event, and associating multiple elements related to the hot security event to perform data fusion.

[0014] With reference to the first aspect, in a third possible implementation manner of the first aspect, after the word corresponding to the end of the entity is pointed to, an entity boundary is determined, different entity boundaries are isolated by using an attribute encryption algorithm, access control of the different entity boundaries is implemented, and the entity boundaries are queried and judged to alarm.

[0015] In a second aspect, the present application provides a threat intelligence analysis system based on a knowledge graph, and the system includes:

[0016] A collecting module is configured to collect state information, domain name information, link addresses and packet data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data;

[0017] An initialization module is configured to perform initialization processing on the open source threat intelligence data, perform text classification on the open source threat intelligence data by using a support vector machine algorithm based on topics, keywords and lengths as features, filter noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences to automatically label identification information to obtain a threat intelligence library.

[0018] A transit module is configured to extract sentences from the threat intelligence library in a predetermined order and input the sentences into an entity extraction module.

[0019] The entity extraction module is configured to input the affiliated beginning word, the entity subsequent word and the non-entity word into different embedding layers according to the label, and input the processed words into corresponding LSTM layers, wherein the processing includes dimension reduction detection of the input sentence and addition of an indication of the affiliated beginning word to a word representing an end of the entity, and the indication is directed to the word representing the end of the entity according to a loss function of the dimension reduction detection, and the word representing the end of the entity is obtained by solving an optimal solution of the loss function.

[0020] After the vectors are exchanged between the adjacent LSTM layers, a current hidden vector is calculated, the hidden vector is divided into a forward hidden vector and a backward hidden vector, the forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and the hidden state is input into a decoding layer, the decoding layer introduces a transition probability algorithm of a label, the label is taken as a label item, and a mapping relationship related to the entity is predicted according to a calculated probability value.

[0021] The visualization module is configured to input the mapping relationship related to the entity, display a knowledge graph of threat intelligence entities, query stored data by using a Cypher statement, and provide the knowledge graph to a user for decision-making.

[0022] In a third aspect, the present application provides a threat intelligence analysis system based on a knowledge graph, and the system includes a processor and a memory:

[0023] The memory is configured to store program code and transmit the program code to the processor.

[0024] The processor is configured to execute the method in any one of the four possibilities of the first aspect according to instructions in the program code.

[0025] In a fourth aspect, the present application provides a computer readable storage medium configured to store program code, and the program code is configured to execute the method in any one of the four possibilities of the first aspect.

[0026] Advantages

[0027] The present application provides a threat intelligence analysis method and system based on a knowledge graph, which improves the embedding layer processing of entity extraction on the basis of existing threat intelligence analysis, can accurately label the entity category and boundary, improves the LSTM layer to obtain the hidden state, and then introduces the transition probability algorithm of the label to predict the mapping relationship related to the entity, thereby overcoming the deficiency that the existing knowledge graph cannot obtain implicit knowledge when performing semantic retrieval, and realizing semantic retrieval and reasoning analysis based on the knowledge graph. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0029] Figure 1 A schematic flowchart of the threat intelligence analysis method based on the knowledge graph according to the present application;

[0030] Figure 2 An architecture diagram of the threat intelligence analysis system based on the knowledge graph according to the present application. DETAILED DESCRIPTION

[0031] The preferred embodiments of the present application will be described in detail below with reference to the drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the protection scope of the present application can be more clearly defined.

[0032] Figure 1 A schematic flowchart of the threat intelligence analysis method based on the knowledge graph according to the present application, the method comprising:

[0033] Collecting state information, domain name information, link addresses and message data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data;

[0034] Performing initialization processing on the open source threat intelligence data, performing text classification on the open source threat intelligence data based on topics, keywords and lengths as features using a support vector machine algorithm, filtering out noise data in the open source threat intelligence data, and performing segmentation processing in units of sentences, automatically labeling identification information, and obtaining a threat intelligence library;

[0035] Extracting sentences from the threat intelligence library in a predetermined order and inputting the sentences into an entity extraction module;

[0036] The entity extraction module sends the affiliated first word, entity subsequent word and non-entity word to different embedding layers according to the labeling, sends the processed words to the corresponding LSTM layer, the processing includes reducing the dimension of the input sentences and adding a guide to the first word, pointing to the word indicating the end of the entity, the pointing is based on the loss function of the dimension reduction detection, and the word indicating the end of the entity is obtained by solving the optimal solution of the loss function.

[0037] After the vectors are exchanged with each other through the adjacent LSTM layers, a current hidden vector is calculated, the hidden vector is divided into a forward hidden vector and a backward hidden vector, the forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and the hidden state is sent to a decoding layer, the decoding layer introduces a transition probability algorithm of a label, takes the label as a label item, and predicts a mapping relationship of the related entity according to a calculated probability value;

[0038] The mapping relationship of the related entity is input into a visualization module to display a knowledge graph of threat intelligence entities, Cypher statements are used to query stored data, and the knowledge graph is provided to a user for decision-making.

[0039] In some preferred embodiments, the collection includes giving different scores to different information sources according to historical records of the information sources, and includes focusing on collecting information corresponding to a pre-set intelligence type, dynamically setting information with low relevance to the intelligence type as redundant information, and removing the redundant information in the initialization processing.

[0040] In some preferred embodiments, the collection includes extracting elements, judging whether the discovered elements are related to a current popular security event, marking a popular security event summary in the elements if yes, and associating multiple elements related to the popular security event to perform data fusion.

[0041] The popular security event includes one or more of a botnet and an attack, the collected popular security event information is subjected to deep correlation analysis and data mining from multiple dimensions of time and space, a rule library is established, suspected attack trace information is compared with information in the rule library, a trace graph is constructed through propagation query and trace query, an occurrence context and an attack path of an attack event are obtained according to the trace graph, and a popular security event summary is marked on the occurrence context.

[0042] In some preferred embodiments, after the word corresponding to the end of the entity is determined, an entity boundary is determined, different entity boundaries are isolated by using an attribute encryption algorithm, access control of different entity boundaries is realized, and query and alarm judgment are performed according to the entity boundaries.

[0043] The method further includes using access control and intrusion detection to perform security audit on important network nodes, network boundaries, and remote access user behaviors, using a time stamp or a counter and combining integrity checking to check the freshness of on-site device authentication data and whether the data is tampered.

[0044] In some preferred embodiments, the clustering algorithm that can be used for data fusion includes a K-Means algorithm, a mean shift clustering algorithm, a density-based clustering algorithm, or a hierarchical clustering algorithm.

[0045] In some preferred embodiments, the visualization display further comprises risk assessment, attack correlation analysis, situation awareness, active defense, data mining in cloud servers, big data analysis, positioning of network vulnerabilities and discovery of potential threats and attacks.

[0046] Figure 2 The architecture diagram of the threat intelligence analysis system based on the knowledge graph provided in the present application, the system comprises:

[0047] The acquisition module is configured to acquire state information, domain name information, link addresses and message data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data;

[0048] The initialization module is configured to perform initialization processing on the open source threat intelligence data, perform text classification based on topics, keywords and lengths as features using a support vector machine algorithm, filter out noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences, automatically label identification information, and obtain a threat intelligence library;

[0049] The transit module is configured to extract sentences from the threat intelligence library in a predetermined order and input them into an entity extraction module;

[0050] The entity extraction module is configured to send the affiliated first word, entity subsequent word and non-entity word to different embedding layers according to the label, and send them to the corresponding LSTM layer after processing, the processing including reducing the dimension of the input sentence and adding an index to the first word, pointing to a word representing the end of the entity, the pointing being based on a loss function of the dimension reduction detection, and the word representing the end of the entity being obtained by solving the optimal solution of the loss function;

[0051] After the adjacent LSTM layers exchange vectors with each other, a current hidden vector is calculated, the hidden vector is divided into a forward hidden vector and a backward hidden vector, the forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and the hidden state is sent to a decoding layer, the decoding layer introduces a label transition probability algorithm, takes the label as a label item, and predicts a mapping relationship related to the entity according to the calculated probability value;

[0052] The visualization module is configured to record the mapping relationship related to the entity, display a knowledge graph of threat intelligence entities, query stored data using a Cypher statement, and provide the knowledge graph to a user for decision-making.

[0053] The present application provides a threat intelligence analysis system based on a knowledge graph, the system comprising a processor and a memory:

[0054] The memory is configured to store program code and transmit the program code to the processor.

[0055] The processor is configured to execute the method according to the instructions in the program code.

[0056] The present application provides a computer readable storage medium configured to store program code, the program code being configured to execute the method according to any one of the embodiments of the first aspect.

[0057] In specific implementations, the present application further provides a computer storage medium, wherein the computer storage medium can store a program, and the program can include some or all steps in the embodiments of the present application when executed. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0058] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disc, an optical disc and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server or a network device and the like) execute the method described in the embodiments of the present application or some parts of the embodiments.

[0059] The same or similar parts among the embodiments of the present application can be referred to each other. In particular, for the embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0060] The above-mentioned embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A knowledge graph based threat intelligence analysis method, characterized in that, The method comprises: Collecting state information, domain name information, link addresses and message data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data; Initializing the open source threat intelligence data, performing text classification on the open source threat intelligence data based on topics, keywords and lengths as features using a support vector machine algorithm, filtering out noise data in the open source threat intelligence data, and performing segmentation processing in units of sentences, automatically labeling identification information, and obtaining a threat intelligence library; Extracting sentences from the threat intelligence library in a predetermined order and inputting the sentences into an entity extraction module; The entity extraction module inputs the initial words, subsequent entity words and non-entity words belonging to different embedding layers according to the labels, and inputs the processed words into corresponding LSTM layers, the processing includes reducing the dimension of the input sentences and adding a guide to the initial words belonging to the same embedding layer, pointing to the words representing the end of the entity, and the pointing is based on the loss function of the dimension reduction detection, and the words representing the end of the entity are obtained by solving the optimal solution of the loss function; After the words representing the end of the entity are pointed, the entity boundaries are determined, different entity boundaries are isolated by using an attribute encryption algorithm, access control of different entity boundaries is realized, and query and alarm judgment are performed according to the entity boundaries; After the adjacent LSTM layers exchange vectors with each other, the current hidden vectors are calculated, the hidden vectors are divided into forward hidden vectors and backward hidden vectors, the forward hidden vectors and the backward hidden vectors are connected to obtain hidden states, and the hidden states are input into a decoding layer, the decoding layer introduces a label transition probability algorithm, takes the labels as label items, and predicts the mapping relationship of the related entities according to the calculated probability values. The mapping relationship of the related entities is input into a visualization module to display a knowledge graph of threat intelligence entities, Cypher sentences are used to query stored data, and the knowledge graph is provided to a user for decision-making.

2. The method of claim 1, wherein: The collection includes giving different scores to different information sources according to historical records of the information sources, and further includes focusing on collecting information corresponding to a pre-set intelligence type, dynamically setting information with low relevance to the intelligence type as redundant information, and removing the redundant information in the initialization processing.

3. The method of claim 1, wherein: The collection includes extracting elements, judging whether the discovered elements are related to a current hot security event, marking a hot security event summary in the elements if the elements are related to the hot security event, and associating multiple elements related to the hot security event to perform data fusion.

4. A knowledge graph based threat intelligence analysis system, characterized in that, The system comprises: A collection module configured to collect state information, domain name information, link addresses and message data from different sensing devices, transit devices, open source platforms and network side devices as open source threat intelligence data; An initialization module configured to initialize the open source threat intelligence data, perform text classification on the open source threat intelligence data based on topics, keywords and lengths as features using a support vector machine algorithm, filter out noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences, automatically label identification information, and obtain a threat intelligence library; A transit module is configured to extract sentences from the threat intelligence library in a predetermined order and input the extracted sentences into an entity extraction module; The entity extraction module is configured to input the belonging start word, entity subsequent word and non-entity word into different embedding layers according to the annotation, and then input the processed words into corresponding LSTM layers, wherein the processing includes dimension reduction detection of the input sentences and addition of a start word pointing to a word indicating the end of an entity, and the pointing is based on a loss function of the dimension reduction detection, and the word indicating the end of the entity is obtained by solving the optimal solution of the loss function. After the pointing to the word indicating the end of the entity, the entity boundary is determined, different entity boundaries are isolated by using an attribute encryption algorithm, access control of the different entity boundaries is realized, and the entity boundaries are used for query and alarm judgment. After the adjacent LSTM layers exchange vectors with each other, a current hidden vector is calculated, the hidden vector is divided into a forward hidden vector and a backward hidden vector, the forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and the hidden state is input into a decoding layer, wherein the decoding layer introduces a label transition probability algorithm, takes the annotation as a label item, and predicts a mapping relationship of the entity according to a calculated probability value. A visualization module is configured to input the mapping relationship of the entity, display a knowledge graph of threat intelligence entities, query stored data by using a Cypher sentence, and provide the knowledge graph to a user for decision-making. 5.A knowledge graph based threat intelligence analysis system, characterized in that, The system includes a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute instructions in the program code to implement the method of any one of claims 1-3.

6. A computer readable storage medium characterized by, The computer-readable storage medium is configured to store program code, and the program code is used to implement the method of any one of claims 1-3.

Images