Network security NER data augmentation method, electronic device and storage medium based on BERT pre-training model

Abstract

The network security NER data augmentation method, electronic device and storage medium based on the BERT pre-training model belong to the field of network security data processing technology. In order to improve the efficiency and accuracy of network security data, the present invention collects cyberspace security reports to obtain cyberspace security data; uses the BIO method to annotate the collected cyberspace security reports, and obtains text sequences and label sequences as original data sets; constructs a BERT pre-training model; masks the cyberspace security data, and then inputs it into the BERT pre-training model for training to obtain a BERT training model for network security NER data; masks the original data set, and inputs it into the BERT training model for network security NER data, and merges the obtained BERT training results for network security NER data with the original data set obtained in step S2 to obtain an augmented data set.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security data processing, and specifically relates to a network security NER data augmentation method based on a BERT pre-training model, an electronic device and a storage medium. Background Art

[0002] Network security refers to the protection of the hardware, software and data in the network system from damage, change or leakage due to accidental or malicious reasons, the continuous and reliable operation of the system and the uninterrupted network service. During the implementation of network security, data storage terminals are required to save data. The monitoring and storage of network security data usually involve manual participation, which leads to low efficiency of network security data storage. In addition, network security data with security vulnerabilities may be stored in the database due to manual review omissions. Therefore, how to obtain network security data accurately and effectively is a technical problem that needs to be solved at this stage. Summary of the invention

[0003] The problem to be solved by the present invention is to improve the efficiency and accuracy of network security data, and propose a network security NER data augmentation method, electronic device and storage medium based on the BERT pre-training model.

[0004] To achieve the above object, the present invention is implemented through the following technical solutions:

[0005] A network security NER data augmentation method based on a BERT pre-training model includes the following steps:

[0006] S1. Collect cyberspace security reports and obtain cyberspace security data for later use;

[0007] S2. Use the BIO method to annotate the cyberspace security report collected in step S1, and obtain the text sequence and label sequence as the original data set for later use;

[0008] S3. Build a BERT pre-training model;

[0009] S4. Mask the cyberspace security data obtained in step S1, and then input it into the BERT pre-training model for training to obtain a BERT training model for cybersecurity NER data;

[0010] S5. Mask the original data set obtained in step S2, input it into the BERT training model for network security NER data obtained in step S4, output the BERT training result for network security NER data, and merge the BERT training result for network security NER data with the original data set obtained in step S2 to obtain an augmented data set of network security NER data based on the BERT pre-training model.

[0011] Furthermore, step S1 collects cyberspace security reports through web crawlers combined with manual labeling by experts.

[0012] Furthermore, the BIO marking method in step S2 is that each tag is composed of one of three parts, the tag B is the beginning of the entity, I is the inside of the entity, and O is belonging to any entity, and the following marking is performed:

[0013] Set annotation set T = (B-entity type, I-entity type, O);

[0014] Based on the cyberspace security data of step S1, given the input sequence X=(x1,x2,…x i …x n ), where x i is the i-th word;

[0015] Generate a label sequence Z=(z1,z2,…z i …z n ), where z i is the i-th mark, z i ∈T, the expression of BIO rule is:

[0016]

[0017] Furthermore, the specific implementation method of step S3 includes the following steps:

[0018] S3.1. Set the given input sequence X = (x1, x2, ... x n );

[0019] S3.2. Randomly select positions in the input sequence for masking and obtain a set of masked positions Then construct the generated mask sequence Y=(y1,y2,…y n ), where for every i∈M i is the i-th mask, for of y i =x i ;

[0020] S3.3. Download the open source BERT pre-trained model on HuggingFace, accept the mask sequence Y as input, and generate the output sequence Y out =(y out1 ,y out2 ,……y outn ), for the mask part, that is, i∈M, the output is the predicted word distribution y outi =P(yi |Y);

[0021] S3.4.BERT pre-trained model is trained to maximize the predicted probability of the masked position. The loss function uses cross entropy loss, which is expressed as:

[0022]

[0023] Among them, L is the training loss, and the training goal is to find a set of y at the corresponding position in the M set i Minimize losses.

[0024] Furthermore, the specific implementation method of step S4 includes the following steps:

[0025] S4.1. The cyberspace security data obtained in step S1 is first divided into segments. For each segment, each time a sentence is added, the overall length of the input segment is judged. If its length is greater than the maximum length max_length required by the model, the sentence is abandoned and the current text segment is saved;

[0026] S4.2. After dividing the text fragment, perform mask processing. If it is a non-entity part and the 0-1 random number is less than the replacement probability p, it is modified to [mask] mask;

[0027] S4.3. Use the BERT pre-trained model obtained in step S3 to train the replaced masked text fragments, setting Refers to the replaced masked text fragment, with a labeled dataset D train , the replacement probability is p, and the BERT after the domain adaptation training model is G;

[0028] BERT automatically generates three embeddings: word embedding e i , position embedding p i and segment embedding d i ; Add word embedding, position embedding and segment embedding to form the final input representation of the word h i , the expression is:

[0029] h i =e i +p i +d i

[0030] BERT consists of multiple layers of Transformer. Assuming there are L layers, the output of each layer is expressed as Where l = 1, 2, 3...L; the expression of the self-attention mechanism in each layer is:

[0031]

[0032] Among them, W Q , W K , W V are the weight matrices of the query matrix, the key matrix, and the value matrix that can be learned, respectively. k is the scaling factor, Q is the query matrix, K is the key matrix, and V is the value matrix;

[0033] S4.4. During training, the cross entropy loss function is used to measure the difference between the predicted probability distribution and the actual label. i The expression of the cross entropy loss is:

[0034]

[0035] Among them, y true is the true label of the i-th word;

[0036] Then the total loss is the sum of the cross entropy losses of all words, expressed as:

[0037]

[0038] The ultimate training goal is to find a set of parameters that minimize the loss and obtain the BERT training model for network security NER data.

[0039] Furthermore, the specific implementation method of step S5 includes the following steps:

[0040] S5.1. First, initialize the augmented data set, denoted as D synthesis , which is empty;

[0041] S5.2. Mask the original data set obtained in step S2 and traverse each If for x i Its corresponding y i If it is O and the random seed < threshold p, then x i Replaced with mask tag [mask];

[0042] S5.3. Use the BERT training model for network security NER data obtained in step S4 to process the sequence with mask tags to obtain the BERT training results for network security NER data

[0043] S5.4. BERT training results for network security NER data obtained using step S5.3 Replace the original sequence And to D synthesis Add the replaced original sequence to ;

[0044] S5.5. Then merge the original data set obtained in step S2 into D synthesis In this paper, we obtain an augmented dataset of network security NER data based on the BERT pre-training model.

[0045] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a network security NER data augmentation method based on a BERT pre-training model are implemented.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a network security NER data augmentation method based on a BERT pre-training model.

[0047] Beneficial effects of the present invention:

[0048] The present invention discloses a network security NER data augmentation method based on a BERT pre-training model, which introduces a Masked Language Model (MLM) task, which is a pre-training method that helps the model better understand the language context. The content of the MLM task is: in a sentence, a certain percentage of tokens are randomly selected and replaced with [MASK]. Then, a classification model is used to predict what word [MASK] actually is, and the sentence is randomly masked and the prediction result is given.

[0049] The network security NER data augmentation method based on the BERT pre-training model described in the present invention uses a pre-training model for prediction. Since the model has been trained on a large amount of corpus data, it can fully consider the influence of context to generate semantically consistent replacement words and reduce the introduction of noise. In addition, the prediction results of the pre-training model may not be synonyms, but other semantically consistent words, which can further improve the diversity of the text and reduce overfitting.

[0050] The network security NER data augmentation method based on the BERT pre-training model described in the present invention utilizes the reasoning ability of the pre-training model, which is stronger than the previous synonym replacement. Because synonym replacement is limited to synonyms, the pre-training model will replace them with logical words.

[0051] The network security NER data augmentation method based on the BERT pre-training model described in the present invention is an augmentation process. The mask is not added randomly directly but has a step of judgment. First, it is marked as O before random replacement. The advantage of this is that for network security data, many entities cannot be replaced. For example, Windows is the noun of the operating system rather than the window. The continuous pre-training technology is used because BERT itself is oriented to the general field and must be infused with network security field knowledge, so a step of continuous pre-training is required.

[0052] The network security NER data augmentation method based on the BERT pre-training model described in the present invention obtains a data set that is the sum of the original data set and the augmented data set, which is necessarily larger than the original data set, and solves the problem that labeled data is difficult to obtain to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of a network security NER data augmentation method based on a BERT pre-training model described in the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0055] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0056] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 The detailed instructions are as follows:

[0057] Embodiment 1:

[0058] A network security NER data augmentation method based on a BERT pre-training model includes the following steps:

[0059] S1. Collect cyberspace security reports and obtain cyberspace security data for later use;

[0060] Furthermore, step S1 collects cyberspace security reports obtained by web crawlers combined with manual labeling by experts;

[0061] S2. Use the BIO method to annotate the cyberspace security report collected in step S1, and obtain the text sequence and label sequence as the original data set for later use;

[0062] Furthermore, the BIO marking method in step S2 is that each tag is composed of one of three parts, the tag B is the beginning of the entity, I is the inside of the entity, and O is belonging to any entity, and the following marking is performed:

[0063] Set annotation set T = (B-entity type, I-entity type, O);

[0064] Based on the cyberspace security data of step S1, given the input sequence X=(x1,x2,…x i …x n ), where x i is the i-th word;

[0065] Generate a label sequence Z=(z1,z2,…z i …z n ), where z i is the i-th mark, z i ∈T, the expression of BIO rule is:

[0066]

[0067] S3. Build a BERT pre-training model;

[0068] Furthermore, the specific implementation method of step S3 includes the following steps:

[0069] S3.1. Set the given input sequence X = (x1, x2, ... x n );

[0070] S3.2. Randomly select positions in the input sequence for masking and obtain a set of masked positions Then construct the generated mask sequence Y=(y1,y2,…y n ), where for every i∈M i is the i-th mask, for of y i =x i ;

[0071] S3.3. Download the open source BERT pre-trained model on HuggingFace, accept the mask sequence Y as input, and generate the output sequence Y out =(y out1 ,y out2 ,……y outn ), for the mask part, that is, i∈M, the output is the predicted word distribution y outi =P(y i |Y);

[0072] S3.4.BERT pre-trained model is trained to maximize the predicted probability of the masked position. The loss function uses cross entropy loss, which is expressed as:

[0073]

[0074] Among them, L is the training loss, and the training goal is to find a set of y at the corresponding position in the M set i Minimize losses.

[0075] S4. Mask the cyberspace security data obtained in step S1, and then input it into the BERT pre-training model for training to obtain a BERT training model for cybersecurity NER data;

[0076] Furthermore, the specific implementation method of step S4 includes the following steps:

[0077] S4.1. The cyberspace security data obtained in step S1 is first divided into segments. For each segment, each time a sentence is added, the overall length of the input segment is judged. If its length is greater than the maximum length max_length required by the model, the sentence is abandoned and the current text segment is saved;

[0078] S4.2. After dividing the text fragment, perform mask processing. If it is a non-entity part and the 0-1 random number is less than the replacement probability p, it is modified to [mask] mask;

[0079] S4.3. Use the BERT pre-trained model obtained in step S3 to train the replaced masked text fragments, setting Refers to the replaced masked text fragment, with a labeled dataset D train , the replacement probability is p, and the BERT after the domain adaptation training model is G;

[0080] BERT automatically generates three embeddings: word embedding e i , position embedding p i and segment embedding d i; Add word embedding, position embedding and segment embedding to form the final input representation of the word h i , the expression is:

[0081] h i =e i +p i +d i

[0082] BERT consists of multiple layers of Transformer. Assuming there are L layers, the output of each layer is expressed as Where l = 1, 2, 3...L; the expression of the self-attention mechanism in each layer is:

[0083]

[0084] Among them, W Q , W K , W V are the weight matrices of the query matrix, the key matrix, and the value matrix that can be learned, respectively. k is the scaling factor, Q is the query matrix, K is the key matrix, and V is the value matrix;

[0085] Q is used to query which parts of the input sequence need to be focused on, and how to find relevant information in the current step. Each query vector corresponds to an element in the input sequence and represents the question asked when looking for the relationship between the element and other elements.

[0086] K contains information labels associated with each element of the input sequence. Each key vector corresponds to an element in the input sequence and represents the characteristics of that element. When calculating the attention weight, the query vector is dot-producted with the key vector to measure the similarity or correlation between the two.

[0087] V contains the actual value or information of the input sequence. Each value vector corresponds to an element in the input sequence and represents the actual information content of the element. After the attention weights are calculated, they are applied to the Value matrix to obtain the attention-weighted output.

[0088] S4.4. During training, the cross entropy loss function is used to measure the difference between the predicted probability distribution and the actual label. i The expression of the cross entropy loss is:

[0089]

[0090] Among them, y true is the true label of the i-th word;

[0091] Then the total loss is the sum of the cross entropy losses of all words, expressed as:

[0092]

[0093] The final training goal is to find a set of parameters that minimize the loss and obtain the BERT training model for network security NER data;

[0094] S5. Mask the original data set obtained in step S2, input it into the BERT training model for network security NER data obtained in step S4, output the BERT training result for network security NER data, and merge the obtained BERT training result for network security NER data with the original data set obtained in step S2 to obtain an augmented data set of network security NER data based on the BERT pre-training model;

[0095] Furthermore, the specific implementation method of step S5 includes the following steps:

[0096] S5.1. First, initialize the augmented data set, denoted as D synthesis , which is empty;

[0097] S5.2. Mask the original data set obtained in step S2 and traverse each If for x i Its corresponding y i If it is O and the random seed < threshold p, then x i Replaced with mask tag [mask];

[0098] S5.3. Use the BERT training model for network security NER data obtained in step S4 to process the sequence with mask tags to obtain the BERT training results for network security NER data

[0099] S5.4. BERT training results for network security NER data obtained using step S5.3 Replace the original sequence And to D synthesis Add the replaced original sequence to ;

[0100] S5.5. Then merge the original data set obtained in step S2 into D synthesis In this paper, we obtain an augmented dataset of network security NER data based on the BERT pre-training model.

[0101] Furthermore, the network security NER data augmentation dataset based on the BERT pre-training model is fine-tuned. i After passing through multiple layers of Transformer, the output after the Lth layer is transformed through a linear layer. Mapped to the label space, the expression is:

[0102]

[0103] Among them, W and b are the parameters that need to be learned, s i is the label score of the i-th word;

[0104] The training process is to maximize the conditional probability, the expression is:

[0105]

[0106] Among them, P(y i = k|X) means that given an input sequence X, the i-th x i The probability of being predicted as label k; s i,k It represents the score of the i-th word corresponding to label k calculated by the linear layer. The specific calculation process is as follows:

[0107]

[0108] Among them, b k is the bias term of the linear layer, corresponding to the bias of label k, and exp is an exponential function.

[0109] Embodiment 2:

[0110] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a network security NER data augmentation method based on a BERT pre-training model described in Example 1 are implemented.

[0111] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. Furthermore, the processor is used to implement the steps of the above-mentioned network security NER data augmentation method based on the BERT pre-training model when executing the computer program stored in the memory.

[0112] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0113] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0114] Embodiment 3:

[0115] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for augmenting network security NER data based on a BERT pre-training model described in Example 1 is implemented.

[0116] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned network security NER data augmentation method based on the BERT pre-training model can be implemented.

[0117] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0118] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0119] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A network security NER data augmentation method based on the BERT pre-training model, characterized in that: The steps include: S1. Collect cyberspace security reports and obtain cyberspace security data for later use; S2. Use the BIO method to annotate the cyberspace security report collected in step S1, and obtain the text sequence and label sequence as the original data set for later use; S3. Build a BERT pre-training model; S4. Mask the cyberspace security data obtained in step S1, and then input it into the BERT pre-training model for training to obtain a BERT training model for cybersecurity NER data; The specific implementation method of step S4 includes the following steps: S4.

1. The cyberspace security data obtained in step S1 is first divided into segments. For each segment, each time a sentence is added, the overall length of the input segment is judged. If its length is greater than the maximum length max_length required by the model, the sentence is abandoned and the current text segment is saved; S4.

2. After dividing the text fragment, perform mask processing. If it is a non-entity part and the 0-1 random number is less than the replacement probability p, it is modified to [mask] mask; S4.

3. Use the BERT pre-trained model obtained in step S3 to train the replaced masked text fragments, setting Refers to the replaced masked text fragment, with a labeled dataset D train , the replacement probability is p, and the BERT after the domain adaptation training model is G; BERT automatically generates three embeddings: word embedding e i , position embedding p i and segment embedding d i ; Add word embedding, position embedding and segment embedding to form the final input representation of the word h i , the expression is: h i =e i +p i +d i BERT consists of multiple layers of Transformer, with L layers, and the output of each layer is expressed as Where l = 1, 2, 3...L; the expression of the self-attention mechanism in each layer is: Among them, W Q , W K , W V are the weight matrices of the query matrix, the key matrix, and the value matrix that can be learned, respectively. k is the scaling factor, Q is the query matrix, K is the key matrix, and V is the value matrix; S4.

4. During training, the cross entropy loss function is used to measure the difference between the predicted probability distribution and the actual label. i The expression of the cross entropy loss is: in, is the true label of the i-th word; Then the total loss is the sum of the cross entropy losses of all words, expressed as: The final training goal is to find a set of parameters that minimize the loss and obtain the BERT training model for network security NER data; S5. Mask the original data set obtained in step S2, input it into the BERT training model for network security NER data obtained in step S4, output the BERT training result for network security NER data, and merge the BERT training result for network security NER data with the original data set obtained in step S2 to obtain an augmented data set of network security NER data based on the BERT pre-training model.

2. According to claim 1, a network security NER data augmentation method based on a BERT pre-training model is characterized in that: Step S1 collects cyberspace security reports through web crawlers combined with manual labeling by experts.

3. According to claim 2, a network security NER data augmentation method based on a BERT pre-training model is characterized in that: The BIO marking method in step S2 is that each tag is composed of one of three parts, the tag B is the beginning of the entity, I is the inside of the entity, and O belongs to any entity, and the following marking is performed: Set annotation set T = (B-entity type, I-entity type, O); Based on the cyberspace security data of step S1, given the input sequence X=(x1,x 2, …x i …x n ), where x i is the i-th word; Generate the annotation sequence Z=(z1,z 2, …z i …z n ), where z i is the i-th mark, z i ∈T, the expression of BIO rule is:

4. According to claim 3, a network security NER data augmentation method based on a BERT pre-training model is characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Set the given input sequence X = (x1, x 2, ……x n ); S3.

2. Randomly select positions in the input sequence for masking and obtain a set of masked positions Then construct the generated mask sequence Y=(y1,y2,…y n ), where for every i∈M i is the i-th mask, for of y i =x i ; S3.

3. Download the open source BERT pre-trained model on HuggingFace, accept the mask sequence Y as input, and generate the output sequence Y out =(y out1 ,y out2 ,……y outn ), for the mask part, that is, i∈M, the output is the predicted word distribution y outi =P(y i |Y); S3.4.BERT pre-trained model is trained to maximize the predicted probability of the masked position. The loss function uses cross entropy loss, which is expressed as: Among them, L is the training loss, and the training goal is to find a set of y at the corresponding position in the M set i Minimize losses.

5. According to claim 4, a network security NER data augmentation method based on a BERT pre-training model is characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. First, initialize the augmented data set, denoted as D synthesis , which is empty; S5.

2. Mask the original data set obtained in step S2 and traverse each If for x i Its corresponding y i If it is O and the random seed < threshold p, then x i Replaced with mask tag [mask]; S5.

3. Use the BERT training model for network security NER data obtained in step S4 to process the sequence with mask tags to obtain the BERT training results for network security NER data S5.

4. BERT training results for network security NER data obtained using step S5.3 Replace the original sequence And to D synthesis Add the replaced original sequence to ; S5.

5. Then merge the original data set obtained in step S2 into D synthesis In this paper, we obtain an augmented dataset of network security NER data based on the BERT pre-training model.

6. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a network security NER data augmentation method based on a BERT pre-training model as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for network security NER data augmentation based on a BERT pre-training model described in any one of claims 1 to 5 is implemented.

Images