Threat intelligence extraction method and system based on semantic analysis

Abstract

The present invention provides a threat intelligence extraction method and system based on semantic analysis. Based on the existing threat intelligence analysis, the enhanced semantic features are obtained by improving the use of two network structures, and the embedding layer processing of entity extraction is improved, so that the entity categories and boundaries can be accurately marked, and the LSTM layer is improved to obtain the hidden state, and the semantic features corresponding to the hidden state are further obtained, thereby overcoming the problems that the existing relationship extraction usually only uses a single deep learning model and cannot capture complete semantic features, and there are a large number of implicit relationships between entities, and the existing model is difficult to effectively capture the semantic features of the implicit relationships.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a threat intelligence extraction method and system based on semantic analysis. Background Art

[0002] Entity is the most basic element in threat intelligence analysis, which describes specific information related to threats. Threat intelligence extraction is based on the accurate identification and extraction of entities. Entity identification is inseparable from the extraction of relationships. Relationship extraction aims to identify certain semantic relationships between entities from unstructured and semi-structured data. However, existing relationship extraction usually only uses a single deep learning model, which cannot capture complete semantic features. At the same time, there are a large number of implicit relationships between entities, and existing models are difficult to effectively capture the semantic features of implicit relationships.

[0003] Therefore, there is an urgent need for a targeted threat intelligence extraction method and system based on semantic analysis. Summary of the invention

[0004] The purpose of the present invention is to provide a threat intelligence extraction method and system based on semantic analysis. On the basis of the existing threat intelligence analysis, the semantic features enhanced by using two network structures are improved, and the embedding layer processing of entity extraction is improved, so that the entity categories and boundaries can be accurately marked, and the LSTM layer is improved to obtain the hidden state, and the semantic features corresponding to the hidden state are further obtained.

[0005] In a first aspect, the present application provides a threat intelligence extraction method based on semantic analysis, the method comprising:

[0006] Collect status information, domain name information, link addresses, and message data from different sensor devices, transit devices, open source platforms, and network-side devices as open source threat intelligence data;

[0007] Initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, segment the data in sentences, and automatically annotate identification information to obtain a threat intelligence library;

[0008] Extracting sentences from the threat intelligence library in a predetermined order and inputting them into an entity extraction module and a semantic analysis module respectively;

[0009] The semantic analysis module reduces the dimension of the input sentence by words, and maps it one by one with the vectors in the continuous space as the semantic features at the word level, thereby obtaining word features;

[0010] According to the GRU network structure, the forward and backward directions are respectively called twice, the word features are input into the GRU network structure called twice, the output results of the two times are spliced ​​to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information;

[0011] fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features;

[0012] The entity extraction module sends the affiliated beginning words, entity subsequent words and non-entity words to different embedding layers according to the annotations, and sends them to the corresponding LSTM layer after processing, wherein the processing includes reducing the dimension of the input sentence for detection, and adding a guide to the affiliated beginning words to point to the corresponding words indicating the end of the entity, wherein the guide is based on the loss function of the reduced dimension detection, and the word indicating the end of the entity is obtained by solving the optimal solution of the loss function;

[0013] After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state;

[0014] The mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature are entered into a visualization module together to display the knowledge graph of the threat intelligence entity and provide it to the user for extraction according to entity relationship or semantic query.

[0015] In combination with the first aspect, in a first possible implementation of the first aspect, the collection includes giving different scores to different information sources based on the historical records of the information sources; and also includes focusing on collecting information corresponding to the pre-set intelligence type based on the pre-set intelligence type, dynamically setting information with low relevance to the intelligence type as redundant information, and clearing it during the initialization process.

[0016] In combination with the first aspect, in a second possible implementation of the first aspect, the collection includes extracting elements, determining whether the discovered elements are related to current hot security events, and if so, marking the hot security event summary in the elements, and associating multiple elements related to the hot security events to perform data fusion.

[0017] In combination with the first aspect, in a third possible implementation of the first aspect, after pointing to the word corresponding to the end of the entity, the entity boundary is determined, different entity boundaries are isolated using an attribute encryption algorithm, different entity boundary access controls are implemented, and queries and alarms are performed according to the entity boundaries.

[0018] In a second aspect, the present application provides a threat intelligence extraction system based on semantic analysis, the system comprising:

[0019] The collection module is used to collect status information, domain name information, link address and message data from different sensor devices, transfer devices, open source platforms and network side devices as open source threat intelligence data;

[0020] An initialization module is used to initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences, automatically annotate identification information, and obtain a threat intelligence library;

[0021] A transfer module, used to extract sentences from the threat intelligence library in a predetermined order and input them into the entity extraction module and the semantic analysis module respectively;

[0022] A semantic analysis module, used to reduce the dimension of the input sentence by words, and map it one by one with the vectors in the continuous space as word-level semantic features to obtain word features;

[0023] According to the GRU network structure, the forward and backward directions are respectively called twice, the word features are input into the GRU network structure called twice, the output results of the two times are spliced ​​to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information;

[0024] fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features;

[0025] An entity extraction module is used to send the affiliated initial word, entity subsequent word and non-entity word to different embedding layers according to the annotation, and send them to the corresponding LSTM layer after processing. The processing includes reducing the dimension of the input sentence and adding a guide to the affiliated initial word to point to the corresponding word indicating the end of the entity. The guide is based on the loss function of the reduced dimension detection, and the optimal solution of the loss function is solved to obtain the word indicating the end of the entity;

[0026] After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state;

[0027] The visualization module is used to input the mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature together, display the knowledge graph of the threat intelligence entity, and provide it to the user for extraction according to entity relationship or semantic query.

[0028] In a third aspect, the present application provides a threat intelligence extraction system based on semantic analysis, the system comprising a processor and a memory:

[0029] The memory is used to store program code and transmit the program code to the processor;

[0030] The processor is used to execute any one of the four possible methods of the first aspect according to the instructions in the program code.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute any one of the four possible methods in the first aspect.

[0032] Beneficial Effects

[0033] The present invention provides a threat intelligence extraction method and system based on semantic analysis. Based on the existing threat intelligence analysis, the enhanced semantic features are obtained by improving the use of two network structures, and the embedding layer processing of entity extraction is improved, so that the entity categories and boundaries can be accurately marked, and the LSTM layer is improved to obtain the hidden state, and the semantic features corresponding to the hidden state are further obtained, thereby overcoming the problems that the existing relationship extraction usually only uses a single deep learning model and cannot capture complete semantic features, and there are a large number of implicit relationships between entities, and the existing model is difficult to effectively capture the semantic features of the implicit relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 The following is a general flow chart of the threat intelligence extraction method based on semantic analysis of the present invention;

[0036] Figure 2 This is an architecture diagram of the threat intelligence extraction system based on semantic analysis of the present invention. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0038] Figure 1 A general flow chart of the threat intelligence extraction method based on semantic analysis provided in this application, the method comprising:

[0039] Collect status information, domain name information, link addresses, and message data from different sensor devices, transit devices, open source platforms, and network-side devices as open source threat intelligence data;

[0040] Initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, segment the data in sentences, and automatically annotate identification information to obtain a threat intelligence library;

[0041] Extracting sentences from the threat intelligence library in a predetermined order and inputting them into an entity extraction module and a semantic analysis module respectively;

[0042] The semantic analysis module reduces the dimension of the input sentence by words, and maps it one by one with the vectors in the continuous space as the semantic features at the word level, thereby obtaining word features;

[0043] According to the GRU network structure, the forward and backward directions are respectively called twice, the word features are input into the GRU network structure called twice, the output results of the two times are spliced ​​to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information;

[0044] fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features;

[0045] The entity extraction module sends the affiliated beginning words, entity subsequent words and non-entity words to different embedding layers according to the annotations, and sends them to the corresponding LSTM layer after processing, wherein the processing includes reducing the dimension of the input sentence for detection, and adding a guide to the affiliated beginning words to point to the corresponding words indicating the end of the entity, wherein the guide is based on the loss function of the reduced dimension detection, and the word indicating the end of the entity is obtained by solving the optimal solution of the loss function;

[0046] After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state;

[0047] The mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature are entered into a visualization module together to display the knowledge graph of the threat intelligence entity and provide it to the user for extraction according to entity relationship or semantic query.

[0048] In some preferred embodiments, the collection includes giving different scores to different information sources based on the historical records of the information sources; and also includes focusing on collecting information corresponding to the pre-set intelligence type based on the pre-set intelligence type, dynamically setting information with low relevance to the intelligence type as redundant information, and clearing it during the initialization process.

[0049] In some preferred embodiments, the collection includes extracting elements, determining whether the discovered elements are related to current hot security events, and if so, marking the hot security event summary in the elements, and associating multiple elements related to the hot security events to perform data fusion.

[0050] The hot security events include one or more of botnets and attacks. The collected hot security event information is subjected to deep correlation analysis and data mining from multiple dimensions of time and space, a rule base is established, the source tracing information of suspected attacks is compared with the information in the rule base, and a source tracing graph is constructed through propagation query and tracing query. The occurrence context and attack path of the attack event are obtained according to the source tracing graph, and a summary of the hot security events is marked on the occurrence context.

[0051] In some preferred embodiments, after pointing to the word corresponding to the end of the entity, the entity boundary is determined, different entity boundaries are isolated using an attribute encryption algorithm, different entity boundary access controls are implemented, and inquiries and alarms are performed according to the entity boundary.

[0052] The method also includes using access control and intrusion detection to conduct security audits on important network nodes, network boundaries, and remote access user behaviors, and using timestamps or counters combined with integrity checks to verify the freshness of on-site device authentication data and whether the detection data has been tampered with.

[0053] In some preferred embodiments, the clustering algorithms that can be used for the data fusion include K-Means algorithm, mean shift clustering algorithm, density-based clustering algorithm, or agglomerative hierarchical clustering algorithm.

[0054] In some preferred embodiments, the visualization also includes risk assessment, attack correlation analysis, situational awareness, and active defense, cooperating with data mining and big data analysis in the cloud server to locate network vulnerabilities and discover potential threats and attacks.

[0055] Figure 2 The architecture diagram of the threat intelligence extraction system based on semantic analysis provided in this application, the system includes:

[0056] The collection module is used to collect status information, domain name information, link address and message data from different sensor devices, transfer devices, open source platforms and network side devices as open source threat intelligence data;

[0057] An initialization module is used to initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences, automatically annotate identification information, and obtain a threat intelligence library;

[0058] A transfer module, used to extract sentences from the threat intelligence library in a predetermined order and input them into the entity extraction module and the semantic analysis module respectively;

[0059] A semantic analysis module, used to reduce the dimension of the input sentence by words, and map it one by one with the vectors in the continuous space as word-level semantic features to obtain word features;

[0060] According to the GRU network structure, the forward and backward directions are respectively called twice, the word features are input into the GRU network structure called twice, the output results of the two times are spliced ​​to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information;

[0061] fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features;

[0062] An entity extraction module is used to send the affiliated initial word, entity subsequent word and non-entity word to different embedding layers according to the annotation, and send them to the corresponding LSTM layer after processing. The processing includes reducing the dimension of the input sentence and adding a guide to the affiliated initial word to point to the corresponding word indicating the end of the entity. The guide is based on the loss function of the reduced dimension detection, and the optimal solution of the loss function is solved to obtain the word indicating the end of the entity;

[0063] After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state;

[0064] The visualization module is used to input the mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature together, display the knowledge graph of the threat intelligence entity, and provide it to the user for extraction according to entity relationship or semantic query.

[0065] The present application provides a threat intelligence extraction system based on semantic analysis, the system comprising: the system comprising a processor and a memory:

[0066] The memory is used to store program code and transmit the program code to the processor;

[0067] The processor is used to execute the method described in any one of all embodiments of the first aspect according to the instructions in the program code.

[0068] The present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in any one of the embodiments of the first aspect.

[0069] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only storage memory (abbreviated as: ROM) or a random access memory (abbreviated as: RAM), etc.

[0070] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0071] The same and similar parts between the various embodiments of this specification can be referred to each other. In particular, for the embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0072] The above-described embodiments of the present invention do not limit the protection scope of the present invention.

Claims

1. A threat intelligence extraction method based on semantic analysis, characterized in that: The method comprises: Collect status information, domain name information, link addresses, and message data from different sensor devices, transit devices, open source platforms, and network-side devices as open source threat intelligence data; Initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, segment the data in sentences, and automatically annotate identification information to obtain a threat intelligence library; Extracting sentences from the threat intelligence library in a predetermined order and inputting them into an entity extraction module and a semantic analysis module respectively; The semantic analysis module reduces the dimension of the input sentence by words, and maps it one by one with the vectors in the continuous space as the semantic features at the word level, thereby obtaining word features; According to the GRU network structure being called twice in the forward and backward directions respectively, the word features are input into the GRU network structure being called twice, the output results of the two times are concatenated to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information; fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features; The entity extraction module sends the affiliated beginning words, entity subsequent words and non-entity words to different embedding layers according to the annotations, and sends them to the corresponding LSTM layer after processing, wherein the processing includes reducing the dimension of the input sentence for detection, and adding a guide to the affiliated beginning words to point to the corresponding words indicating the end of the entity, wherein the guide is based on the loss function of the reduced dimension detection, and the word indicating the end of the entity is obtained by solving the optimal solution of the loss function; After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state; The mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature are entered into a visualization module together to display the knowledge graph of the threat intelligence entity and provide it to the user for extraction according to entity relationship or semantic query.

2. The method according to claim 1, characterized in that: The collection includes giving different scores to different information sources according to the historical records of the information sources; and also includes focusing on collecting information corresponding to the pre-set intelligence type according to the pre-set intelligence type, dynamically setting information with low relevance to the intelligence type as redundant information, and clearing it during the initialization process.

3. The method according to claim 1, characterized in that: The collection includes extracting elements, determining whether the discovered elements are related to current hot security events, and if so, marking a summary of the hot security event in the elements, and associating multiple elements related to the hot security events to perform data fusion.

4. The method according to any one of claims 2 or 3, characterized in that: After pointing to the word corresponding to the end of the entity, the entity boundary is determined, and the attribute encryption algorithm is used to isolate different entity boundaries, implement different entity boundary access controls, and query and judge alarm according to the entity boundary.

5. A threat intelligence extraction system based on semantic analysis, characterized in that: The system comprises: The collection module is used to collect status information, domain name information, link address and message data from different sensor devices, transfer devices, open source platforms and network side devices as open source threat intelligence data; An initialization module is used to initialize the open source threat intelligence data, use the support vector machine algorithm to perform text classification based on the subject, keyword, and length as features, filter out noise data in the open source threat intelligence data, and perform segmentation processing in units of sentences, automatically annotate identification information, and obtain a threat intelligence library; A transfer module, used to extract sentences from the threat intelligence library in a predetermined order and input them into the entity extraction module and the semantic analysis module respectively; A semantic analysis module, used to reduce the dimension of the input sentence by words, and map it one by one with the vectors in the continuous space as word-level semantic features to obtain word features; According to the GRU network structure, the forward and backward directions are respectively called twice, the word features are input into the GRU network structure called twice, the output results of the two times are spliced ​​to obtain the current global semantic information, and the word features are input into the CNN network structure to obtain the current local semantic information; fusing the global semantic information and the local semantic information in time to obtain enhanced semantic features; An entity extraction module is used to send the affiliated initial word, entity subsequent word and non-entity word to different embedding layers according to the annotation, and send them to the corresponding LSTM layer after processing. The processing includes reducing the dimension of the input sentence and adding a guide to the affiliated initial word to point to the corresponding word indicating the end of the entity. The guide is based on the loss function of the reduced dimension detection, and the optimal solution of the loss function is solved to obtain the word indicating the end of the entity; After the adjacent LSTM layers exchange vectors with each other, the current hidden vector is calculated, and the hidden vector is divided into a forward hidden vector and a backward hidden vector. The forward hidden vector and the backward hidden vector are connected to obtain a hidden state, and then the hidden state is sent to the decoding layer and the semantic analysis module respectively. The decoding layer introduces a label transfer probability algorithm, takes the annotation as a label item, and predicts the mapping relationship of the relevant entities according to the calculated probability value. The semantic analysis module outputs a second semantic feature corresponding to the hidden state; The visualization module is used to input the mapping relationship of the relevant entities, the second semantic feature and the enhanced semantic feature together, display the knowledge graph of the threat intelligence entity, and provide it to the user for extraction according to entity relationship or semantic query.

6. A threat intelligence extraction system based on semantic analysis, characterized in that: The system comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method according to any one of claims 1 to 4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the method according to any one of claims 1 to 4.

Images