Tranform-based network security risk knowledge graph triple joint extraction method

Abstract

The invention discloses a network security risk knowledge graph triple joint extraction method based on Transform. The method specifically comprises the following steps: step 1, constructing a network security knowledge graph ontology oriented to network threat intelligence; step 2, collecting network threat intelligence in the network security field, extracting text data in the network threat intelligence in the network security field, segmenting and labeling the text data according to statements to generate a data set, and dividing the text data into a training set and a data set; step 3, constructing a joint relation triple extraction model; 4, training the constructed joint relation triple extraction model by adopting a training set; 5, inputting the test set into the extraction model, and evaluating the performance of the extraction model; and step 6, performing network security risk knowledge graph triple joint extraction by using the tested model. The problem that in the prior art, the extraction precision of the triad is poor due to the fact that global association of the text data sequence and association between the entity pair and the relation sequence are ignored is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security knowledge graph construction methods, and relates to a Transformer-based network security risk knowledge graph triple joint extraction method. Background Art

[0002] The information source of network security risk is mainly composed of Cyber ​​Threat Intelligence (CTI). CTI refers to a knowledge set that helps researchers better understand, predict and defend against network attacks by collecting and analyzing information about network threats. It plays an important role in improving network security situation awareness and security defense capabilities. At present, the types of network threat intelligence are diverse, the distribution is discrete, and the terminology is not standardized, which poses great obstacles in practical application. As an emerging information organization and processing technology, knowledge graph has received extensive attention and application in the field of network security because it can effectively integrate and manage large-scale, multi-source heterogeneous data and provide powerful knowledge representation and reasoning capabilities. Introducing the knowledge graph technology framework in the study of network security risks, using the idea of ​​information collection and processing integration of knowledge graphs for massive data, and constructing a network security knowledge graph for network threat intelligence can improve the collection efficiency and intelligence quality of network threat intelligence, while expanding the scope of intelligence use.

[0003] Relation triple extraction plays a vital role in the automatic construction of knowledge graphs. Relation triple extraction refers to extracting triple information with clear semantic structure from text data and constructing triples (s, r, o), where s represents the subject, o represents the object, and r represents the relationship between the two entities. In recent years, with the application of deep learning technology, the relationship triple extraction method has made significant progress. The relationship triple joint extraction method can effectively avoid error propagation by simultaneously encoding and decoding entities and relations, and can better identify and handle the overlap of entities and relations.

[0004] Some of the latest methods for joint extraction of relational triples have demonstrated strong extraction capabilities on different datasets, especially the ability to extract triples from complex sentences containing overlapping or multiple triples. However, existing methods mainly extract triples based on local features extracted from single entity pairs or limited entity pairs, ignoring the following two valuable global features: the global association of text data sequences, and the association between entity pairs and relation sequences. These two associations can reveal the differences and connections between entities and between entities and relations in text data, and help improve the accuracy of relational triple extraction. Therefore, existing methods ignore the global association of text data sequences, the association between entity pairs and relation sequences, resulting in poor accuracy of triple extraction. Summary of the invention

[0005] The purpose of the present invention is to provide a Transformer-based network security risk knowledge graph triple joint extraction method, which solves the problem that the prior art ignores the global association of text data sequences and the association between entity pairs and relationship sequences, resulting in poor triple extraction accuracy.

[0006] The technical solution adopted by the present invention is a method for jointly extracting triples of network security risk knowledge graph based on Transformer, which is specifically implemented in the following steps:

[0007] Step 1: Construct a network security knowledge graph ontology for network threat intelligence;

[0008] Step 2: Collect network threat intelligence in the field of network security, extract text data from the network threat intelligence in the field of network security and segment it into several sentences by sentence, annotate each sentence based on the network security knowledge graph ontology constructed in step 1 to generate a data set, and divide it into a training set and a data set;

[0009] Step 3: Construct the joint relation triple extraction model GAIRel;

[0010] Step 4, using the training set to train the joint relation triple extraction model GAIRel constructed in step 3 to obtain an extraction model;

[0011] Step 5: Input the test set into the extraction model to evaluate the performance of the extraction model;

[0012] Step 6: Use the tested model to jointly extract triples from the cybersecurity risk knowledge graph.

[0013] The present invention is also characterized in that:

[0014] Step 1 is as follows:

[0015] Based on the STIX standardized language, a network security knowledge graph ontology for network threat intelligence is constructed. The network security knowledge graph ontology includes entity types, semantic relationship types, and semantic relationships between entities.

[0016] Step 2 is as follows: collect APT reports, security bulletins, and vulnerability libraries in network threat intelligence in the field of network security, and extract text data from APT reports, security bulletins, and vulnerability libraries, then perform semantic segmentation on the extracted text data and divide it into several sentences, and then annotate each sentence based on the network security knowledge graph ontology constructed in step 1 to generate a data set, which is divided into a training set and a data set;

[0017] Among them, the specific annotation of each sentence is as follows:

[0018] Analyze each sentence according to the entity type, semantic relationship type, and semantic relationship between entities determined in step 1, extract all relationship triplets in the sentence and label them.

[0019] The joint relation triple extraction model GAIRel in step 3 includes an encoder, a global interaction module, a feature extraction module, and a decoder connected in sequence;

[0020] Use the pre-trained BERT-Base-Cased model as the encoder;

[0021] The global interaction module consists of two independent bidirectional long short-term memory network layers followed by two independent feed-forward layers;

[0022] The feature extraction module includes a subject-object feature interaction layer, a maximum pooling layer, two multi-head attention layers, and layer normalization;

[0023] The decoding module uses the table filling strategy to generate the relationship table and entity table and decode the final extraction result of the model for the relationship triples in the sentence.

[0024] Step 4 is as follows:

[0025] Step 4.1: Input the sentences in the training set into the encoder. The pre-trained BERT-Base-Cased model is used as the encoder to encode the sentences into a token sequence, denoted as Where n is the number of words in the sentence, d h is the vector dimension;

[0026] Step 4.2: Input the token sequence H into the global interaction module to obtain the subject feature vector and the object feature vector, which are recorded as and

[0027] Step 4.3, and Input it into the feature extraction module and perform N iterations to obtain the extracted feature vector

[0028] Step 4.4, transform the feature vector The input is sent to the decoding module, which uses the table filling strategy to generate the relationship table and entity table and decode the final extraction results of the relationship triples in the sentence.

[0029] Step 4.2 is as follows:

[0030] The token sequence H is input into two bidirectional long short-term memory network layers to generate subject-related feature vectors The feature vector associated with the object Specifically:

[0031]

[0032] Then and Input the two feedforward layers respectively to perform feature conversion, as shown in formula (2):

[0033]

[0034] in, and are the training weights and biases respectively.

[0035] Step 4.3 is as follows:

[0036] Step 4.3.1: The outputs of the two feed-forward layers and As input, it is input into the subject-object feature interaction layer to generate relationship features through the interaction of two feature vectors

[0037] Step 4.3.2: Relationship features Input to the maximum pooling layer to generate a feature vector related to the subject and the feature vector F associated with the object o (1) ;

[0038] Step 4.3.3, the feature vector F associated with the subject s (1) and the feature vector F associated with the object o (1) Input to the first multi-head self-attention layer to obtain features in, include and in is the feature vector associated with the subject The features obtained by inputting into the first multi-head self-attention layer, Denotes the feature vector F associated with the object o (1) Input the features obtained by the first multi-head self-attention layer;

[0039] Step 4.3.4, the features And the token sequence H obtained in step 4.1 is input into the second multi-head self-attention layer to obtain in, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features;

[0040] Step 4.3.5, Perform layer normalization operation to get the output in, include and in, For the general After layer normalization, the features are obtained. For the general Features obtained after layer normalization operation;

[0041] Step 4.3.6: Add the That is and Input it into the subject-object feature interaction layer and continue to iterate for the second time according to steps 4.3.1-4.3.5 to obtain Then Input it into the subject-object feature interaction layer and continue to iterate for the third time according to steps 4.3.1-4.3.5 to obtain This cycle iterates until we get Where N is the total number of iterations.

[0042] In step 4.3.1, for the nth iteration, the subject-object feature interaction layer performs the interaction of two feature vectors to generate the relationship feature F r (n) , as shown in formula (3):

[0043]

[0044] where · represents the Hamada product, and The input of the subject-object feature interaction layer at the nth iteration, that is, the output after the layer normalization operation at the n-1th iteration, and They are training weights and biases respectively;

[0045] In step 4.3.2, for the nth iteration, the relation feature F r (n) Enter the maximum pooling layer, the operation of the maximum pooling layer is as follows:

[0046]

[0047] in, are training weights and biases respectively;

[0048] In steps 4.3.3-4.3.5, for the nth iteration, the operation of the first layer of multi-head self-attention layer and the operation of the second layer of multi-head self-attention layer and the layer normalization operation are as follows:

[0049]

[0050] in, is the output of the first multi-head self-attention layer in the nth iteration, is the output of the second multi-head self-attention layer in the nth iteration, is the output after the normalization operation of the middle layer in the nth iteration, where include and Including F s (n) and F o (n) , include and and In the nth iteration, F s (n) and F o (n) The output obtained after inputting the first layer of multi-head self-attention layer, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features; include and and Respectively for and The features obtained after layer normalization are the final output of the nth iteration;

[0051] The last iteration output include and

[0052] Step 4.4 is as follows:

[0053] The decoding module outputs the final result in step 4.3 and As input, firstly, the subject feature vector and the object feature vector are interacted to generate the total table feature vector TF t (N) , as shown in formula (6):

[0054]

[0055] where · represents the Hamada product, and are training weights and biases respectively;

[0056] Then, according to TF t (N) Generate relationship table and entity table, as shown in formula (7):

[0057]

[0058] Among them, TF i Represents the initial entity table feature, TF r Indicates the initial relational table characteristics, table i With table r It represents the final generated result of the entity table and relationship table obtained by the final decoding, L i , L r Label the entity table and relationship table respectively. are training weights and biases respectively;

[0059] The entity table is a two-dimensional matrix that indicates the subject and object that appear in the sentence; the relationship table is a three-dimensional matrix, and each relationship corresponds to a two-dimensional matrix. The relationship triples in the sentence are decoded according to the entities marked in the entity table.

[0060] Step 5 is as follows:

[0061] Input the test data set established in step 2 into the relation triple joint extraction model l trained in step 4 for extraction test, and evaluate the extraction model by using accuracy, recall rate, and F1-score;

[0062] Step 6 specifically includes: performing sentence segmentation on the text data to be detected into several sentences, and inputting each sentence into the extraction model evaluated in step 5 to obtain the extraction result.

[0063] The beneficial effects of the present invention are:

[0064] Based on the idea of ​​Transformer, this paper constructs a joint extraction model for relation triples in knowledge graphs. It utilizes two valuable global features, namely the global association of text data sequences and the association between entity pairs and relation sequences, to highlight the differences and connections between entities and between entities and relations in text data, thereby improving the accuracy of knowledge graph relation triple extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of the method for jointly extracting triples of network security risk knowledge graph based on Transformer of the present invention;

[0066] Figure 2 It is a semantic relationship link diagram between entities in the Transformer-based triple joint extraction method of network security risk knowledge graph of the present invention;

[0067] Figure 3 It is a joint relationship triple extraction model diagram constructed in the Transformer-based network security risk knowledge graph triple joint extraction method of the present invention. DETAILED DESCRIPTION

[0068] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Example 1

[0070] The present invention is based on the Transformer network security risk knowledge graph triple joint extraction method, the process is as follows Figure 1 As shown, the specific steps are as follows:

[0071] Step 1: Construct a network security knowledge graph ontology for network threat intelligence. Specifically, based on the STIX standardized language, construct a network security knowledge graph ontology for network threat intelligence. The network security knowledge graph ontology includes entity types, semantic relationship types, and semantic relationships between entities.

[0072] Define 12 types of entities: attack mode, attack activity, countermeasures, identity, infrastructure, assets, malware, impact, report, threat subject, tool, vulnerability; define 10 semantic relationships: exploit, target, penetrate, attribute, mitigate, remediate, own, host, forge, and deploy; semantic relationship links between entities are as follows: Figure 2 As shown;

[0073] Step 2: Collect network threat intelligence in the field of network security, extract text data from network threat intelligence in the field of network security and segment it into several sentences by sentence. Based on the network security knowledge graph ontology constructed in step 1, annotate each sentence to generate a data set, and divide it into a training set and a data set, specifically:

[0074] Collect APT reports, security bulletins, and vulnerability libraries in the network threat intelligence in the field of network security, and extract text data from APT reports, security bulletins, and vulnerability libraries, then perform semantic segmentation on the extracted text data and divide it into several sentences, and then annotate each sentence based on the network security knowledge graph ontology constructed in step 1 to generate a data set, and divide it into a training set and a data set; wherein, annotating each sentence specifically includes: analyzing each sentence according to the entity type, semantic relationship type, and semantic relationship between entities determined in step 1, extracting and annotating all relationship triples in the sentence.

[0075] Step 3: Construct the joint relation triple extraction model GAIRel, as Figure 3 As shown, it includes an encoder, a global interaction module, a feature extraction module, and a decoder connected in sequence;

[0076] Use the pre-trained BERT-Base-Cased model as the encoder;

[0077] The global interaction module consists of two independent bidirectional long short-term memory network layers followed by two independent feed-forward layers;

[0078] The feature extraction module includes a subject-object feature interaction layer, a maximum pooling layer, two multi-head attention layers, and layer normalization;

[0079] The decoding module uses the table filling strategy to generate the relationship table and entity table and decode the final extraction result of the model for the relationship triples in the sentence;

[0080] Step 4: Use the training set to train the joint relation triple extraction model GAIRel constructed in step 3 to obtain the extraction model, which is specifically:

[0081] Step 4.1: Input the sentences in the training set into the encoder. The pre-trained BERT-Base-Cased model is used as the encoder to encode the sentences into a token sequence, denoted as Where n is the number of words in the sentence, d h is the vector dimension;

[0082] Step 4.2: Input the token sequence H into the global interaction module to obtain the subject feature vector and the object feature vector, which are recorded as and Specifically:

[0083] The token sequence H is input into two bidirectional long short-term memory network layers. The bidirectional long short-term memory network layer enables the subject, object and token sequence H to interact, associate feature vectors with each other, and extract useful feature information of the global association between entity pairs and text data sequences, so that the model can determine the relationship type of entity pairs more quickly and generate feature vectors related to the subject respectively. The feature vector associated with the object Specifically:

[0084]

[0085] Then and Input the two feedforward layers respectively to perform feature conversion, as shown in formula (2):

[0086]

[0087] in, and They are training weights and biases respectively;

[0088] Step 4.3, and Input it into the feature extraction module and perform N iterations to obtain the extracted feature vector Specifically:

[0089] Step 4.3.1: The outputs of the two feed-forward layers and As input, it is input into the subject-object feature interaction layer to generate the relationship feature F through the interaction of the two feature vectors r (1) ;

[0090] Step 4.3.2, the relationship feature F r (1) Input to the maximum pooling layer to generate the feature vector F related to the subject s (1) and the feature vector F associated with the object o (1) ;

[0091] Step 4.3.3, the feature vector F associated with the subject s (1) and the feature vector associated with the object Input to the first multi-head self-attention layer to obtain features in, include and in is the feature vector F associated with the subject s(1) The features obtained by inputting into the first multi-head self-attention layer, Denotes the feature vector F associated with the object o (1) Input the features obtained by the first multi-head self-attention layer;

[0092] Step 4.3.4, the features And the token sequence H obtained in step 4.1 is input into the second multi-head self-attention layer to obtain in, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features;

[0093] Step 4.3.5, Perform layer normalization operation to get the output in, include and in, For the general After layer normalization, the features are obtained. For the general Features obtained after layer normalization operation;

[0094] Step 4.3.6: Add the That is and Input it into the subject-object feature interaction layer and continue to iterate for the second time according to steps 4.3.1-4.3.5 to obtain Then Input it into the subject-object feature interaction layer and continue to iterate for the third time according to steps 4.3.1-4.3.5 to obtain This cycle iterates until we get Where N is the total number of iterations;

[0095] Among them, in step 4.3.1, for the nth iteration, the subject-object feature interaction layer performs the interaction of two feature vectors to generate the relationship feature F r (n) , as shown in formula (3):

[0096]

[0097] where · represents the Hamada product, and The input of the subject-object feature interaction layer at the nth iteration, that is, the output after the layer normalization operation at the n-1th iteration, and They are training weights and biases respectively;

[0098] In step 4.3.2, for the nth iteration, the relation feature F r (n) Input the maximum pooling layer to generate the feature vector F related to the subject s (n) and the feature vector F associated with the object o (n) , the operation of the maximum pooling layer is as follows:

[0099]

[0100] in, are training weights and biases respectively;

[0101] In steps 4.3.3-4.3.5, the idea of ​​the Transformer decoder is used to mine the global associations between the subject, relation, and object in the sentence, including continuous multi-head self-attention layers, multi-head attention layers, and layer normalization;

[0102] The multi-head self-attention layer helps the model capture richer feature information and explore the relationship between the subject and the object in more depth;

[0103] After the multi-head self-attention layer, a multi-head attention layer is connected, and H in step 4.1 is also used as part of the input. The interaction between the subject and object feature vectors and the overall feature H of the sentence can discover the hidden global feature associations between the subject and the sentence, and between the object and the sentence; then the output To avoid the gradient vanishing problem that may occur in the feature extraction module due to multiple iterations, this step uses and As input, residual connections are used to generate the final feature vectors between subjects, relations, and objects. and The details are as follows:

[0104] In steps 4.3.3-4.3.5, for the nth iteration, the operation of the first layer of multi-head self-attention layer and the operation of the second layer of multi-head self-attention layer and the layer normalization operation are as follows:

[0105]

[0106] in, is the output of the first multi-head self-attention layer in the nth iteration, is the output of the second multi-head self-attention layer in the nth iteration, is the output after the normalization operation of the middle layer in the nth iteration, where include and Including F s (n) and F o (n) , include and and In the nth iteration, F s (n) and F o (n) The output obtained after inputting the first layer of multi-head self-attention layer, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features; include and and Respectively for and The features obtained after layer normalization are the final output of the nth iteration;

[0107] The last iteration output include and

[0108] Step 4.4, transform the feature vector The input is sent to the decoding module, which uses the table filling strategy to generate the relationship table and entity table and decode the final extraction result of the relationship triples in the sentence; specifically:

[0109] The decoding module outputs the final result in step 4.3 and As input, firstly, the subject feature vector and the object feature vector are interacted to generate the total table feature vector TF t (N) , as shown in formula (6):

[0110]

[0111] where · represents the Hamada product, and are training weights and biases respectively;

[0112] Then, according to TF t (N) Generate relationship table and entity table, as shown in formula (7):

[0113]

[0114] Among them, TF i Represents the initial entity table feature, TF r Indicates the initial relational table characteristics, table i With table r It represents the final generated result of the entity table and relationship table obtained by the final decoding, L i , L r are the annotation sets of entity tables and relationship tables respectively, W i , W r ∈R dh×dh 、b i 、b r ∈R dh are training weights and biases respectively;

[0115] The entity table is a two-dimensional matrix that indicates the subject and object that appear in the sentence; the relationship table is a three-dimensional matrix, and each relationship corresponds to a two-dimensional matrix. The relationship triples in the sentence are decoded according to the entities marked in the entity table.

[0116] Step 5: Input the test set into the extraction model to evaluate the performance of the extraction model. Input the test data set established in step 2 into the relation triple joint extraction model l trained in step 4 for extraction testing. The extraction model is evaluated by using accuracy, recall rate, and F1-score.

[0117] Step 6: Use the tested model to jointly extract triples from the cybersecurity risk knowledge graph. Specifically, perform sentence segmentation on the text data to be tested into several sentences, and input each sentence into the extraction model evaluated in step 5 to obtain the extraction result.

[0118] In this embodiment, a grid search method is used to optimize the hyper parameters in the relation triple joint extraction model GAIRel: learning rate, Dropout percentage, activation function, and optimizer for calculating loss to minimize the loss function.

[0119] Example 2

[0120] Based on Example 1, some neurons are randomly dropped according to the Dropout percentage during the training process to reduce the risk of overfitting. The dropout ratio is set to between 0.2 and 0.5. After the model training is completed, the Dropout percentage is set to zero to ensure the accuracy of the model in testing or actual applications.

[0121] Example 3

[0122] Based on Example 2, in step 2, 90% of the data in the data set is used as a training set, and 10% of the data in the data set is used as a test set.

[0123] Example 4

[0124] Based on Example 3, this example selects the public data sets WebNLG_star and NYT for testing. The data set division and test results are as follows:

[0125] Table 1 Dataset details

[0126] Dataset Training set Validation set Test Set Number of relationships WebNLG_star 5019 500 703 171 NYT 56195 5000 5000 24

[0127] Table 2 Model evaluation indicators

[0128] Dataset Accuracy Recall F1 score Sample size WebNLG_star validation set 0.94364 0.93261 0.93809 500 WebNLG_star test set 0.93610 0.92663 0.93134 703 NYT validation set 0.91933 0.92576 0.91299 5000 NYT test set 0.91813 0.91532 0.92096 5000

[0129] The test results are shown in Table 2. From the results in Table 2, it can be seen that the triple extraction accuracy, recall rate, F1 score, etc. of the method of the present invention are all at a high level, and the accuracy of the model is high, which can prove that the present invention is effective in the joint extraction of relation triples.

Claims

1. A Transformer-based triple extraction method for network security risk knowledge graph, characterized by: The specific steps are as follows: Step 1: Construct a network security knowledge graph ontology for network threat intelligence; Step 2: Collect network threat intelligence in the field of network security, extract text data from the network threat intelligence in the field of network security and segment it into several sentences by sentence, annotate each sentence based on the network security knowledge graph ontology constructed in step 1 to generate a data set, and divide it into a training set and a data set; Step 3: Construct the joint relation triple extraction model GAIRel; Step 4, using the training set to train the joint relation triple extraction model GAIRel constructed in step 3 to obtain an extraction model; Step 5: Input the test set into the extraction model to evaluate the performance of the extraction model; Step 6: Use the tested model to jointly extract triples from the cybersecurity risk knowledge graph.

2. According to the Transformer-based network security risk knowledge graph triple joint extraction method of claim 1, it is characterized in that: The step 1 is specifically as follows: Based on the STIX standardized language, a network security knowledge graph ontology for network threat intelligence is constructed. The network security knowledge graph ontology includes entity types, semantic relationship types, and semantic relationships between entities.

3. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 2 is characterized in that: The step 2 specifically includes: collecting APT reports, security announcements, and vulnerability libraries in the network threat intelligence in the network security field, and extracting text data in the APT reports, security announcements, and vulnerability libraries, and then performing semantic segmentation on the extracted text data to divide it into several sentences, and then annotating each sentence based on the network security knowledge graph ontology constructed in step 1 to generate a data set, and dividing it into a training set and a data set; Among them, the specific annotation of each sentence is as follows: Analyze each sentence according to the entity type, semantic relationship type, and semantic relationship between entities determined in step 1, extract all relationship triplets in the sentence and label them.

4. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 3 is characterized in that: The joint relation triple extraction model GAIRel in step 3 includes an encoder, a global interaction module, a feature extraction module, and a decoder connected in sequence; Use the pre-trained BERT-Base-Cased model as the encoder; The global interaction module is composed of two independent bidirectional long short-term memory network layers and two independent feed-forward layers thereafter; The feature extraction module includes a subject-object feature interaction layer, a maximum pooling layer, two multi-head attention layers, and layer normalization; The decoding module uses a table filling strategy to generate a relationship table and an entity table and decodes the final extraction result of the model for the relationship triples in the sentence.

5. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 4 is characterized in that: The step 4 is specifically as follows: Step 4.1: Input the sentences in the training set into the encoder. The pre-trained BERT-Base-Cased model is used as the encoder to encode the sentences into a token sequence, denoted as Where n is the number of words in the sentence, d h is the vector dimension; Step 4.2: Input the token sequence H into the global interaction module to obtain the subject feature vector and the object feature vector, which are recorded as and Step 4.3, and Input it into the feature extraction module and perform N iterations to obtain the extracted feature vector Step 4.4, transform the feature vector The input is sent to the decoding module, which uses the table filling strategy to generate the relationship table and entity table and decode the final extraction results of the relationship triples in the sentence.

6. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 5 is characterized in that: The step 4.2 is specifically as follows: The token sequence H is input into two bidirectional long short-term memory network layers to generate subject-related feature vectors The feature vector associated with the object Specifically: Then and Input the two feedforward layers respectively to perform feature conversion, as shown in formula (2): in, and are the training weights and biases respectively.

7. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 6 is characterized in that: The step 4.3 is specifically as follows: Step 4.3.1: The outputs of the two feed-forward layers and As input, it is input into the subject-object feature interaction layer to generate relationship features through the interaction of two feature vectors Step 4.3.2: Relationship features Input to the maximum pooling layer to generate a feature vector related to the subject and the feature vector associated with the object Step 4.3.3, the feature vector associated with the subject and the feature vector associated with the object Input to the first multi-head self-attention layer to obtain features in, include and in is the feature vector associated with the subject The features obtained by inputting into the first multi-head self-attention layer, Represents the feature vector associated with the object Input the features obtained by the first multi-head self-attention layer; Step 4.3.4, the features And the token sequence H obtained in step 4.1 is input into the second multi-head self-attention layer to obtain in, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features; Step 4.3.5, Perform layer normalization operation to get the output in, include and in, For the general After layer normalization, the features are obtained. For the general Features obtained after layer normalization operation; Step 4.3.6: Add the That is and Input it into the subject-object feature interaction layer and continue to iterate for the second time according to steps 4.3.1-4.3.5 to obtain Then Input it into the subject-object feature interaction layer and continue to iterate for the third time according to steps 4.3.1-4.3.5 to obtain ...., and repeat this cycle until you get Where N is the total number of iterations.

8. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 7 is characterized in that: In step 4.3.1, for the nth iteration, the subject-object feature interaction layer generates the relationship feature by interacting the two feature vectors As shown in formula (3): where · represents the Hamada product, and The input of the subject-object feature interaction layer at the nth iteration, that is, the output after the layer normalization operation at the n-1th iteration, and They are training weights and biases respectively; In step 4.3.2, for the nth iteration, the relationship feature Enter the maximum pooling layer, the operation of the maximum pooling layer is as follows: in, are training weights and biases respectively; In steps 4.3.3-4.3.5, for the nth iteration, the operation of the first layer of multi-head self-attention layer and the operation of the second layer of multi-head self-attention layer and the layer normalization operation are as follows: in, is the output of the first multi-head self-attention layer in the nth iteration, is the output of the second multi-head self-attention layer in the nth iteration, is the output after the normalization operation in the nth iteration, where include and include and include and and In the nth iteration, and The output obtained after inputting the first layer of multi-head self-attention layer, include and For the general And H are input into the second multi-head self-attention layer to obtain features, For the general And H are input into the second multi-head self-attention layer to obtain features; include and and Respectively and The features obtained after layer normalization are the final output of the nth iteration; The last iteration output include and 9. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 8 is characterized in that: The step 4.4 is specifically as follows: The decoding module outputs the final result in step 4.3 and As input, firstly, the subject feature vector and the object feature vector are interacted to generate the total table feature vector As shown in formula (6): where · represents the Hamada product, and are training weights and biases respectively; Then, according to TF t (N) Generate relationship table and entity table, as shown in formula (7): Among them, TF i Represents the initial entity table feature, TF r Indicates the initial relational table characteristics, table i With table r It represents the final generated result of the entity table and relationship table obtained by the final decoding, L i , L r Label the entity table and relationship table respectively. are training weights and biases respectively; The entity table is a two-dimensional matrix that indicates the subject and object that appear in the sentence; the relationship table is a three-dimensional matrix, and each relationship corresponds to a two-dimensional matrix. The relationship triples in the sentence are decoded according to the entities marked in the entity table.

10. The method for jointly extracting triples of network security risk knowledge graph based on Transformer according to claim 9 is characterized in that: The step 5 is specifically as follows: Input the test data set established in step 2 into the relation triple joint extraction model l trained in step 4 for extraction test, and evaluate the extraction model by using accuracy, recall rate, and F1-score; The step 6 specifically includes: performing sentence segmentation on the text data to be detected into a number of sentences, and inputting each sentence into the extraction model evaluated in step 5 to obtain the extraction result.

Images